TEST  AND  EVALUATION 
MANAGEMENT  GUIDE 

MARCH  1988 


DTIC 


DISTRIBUTION  STATEMENT  A 

Approved  for  public  release! 
v  Disfraouuoa  Unlimited 


DEEENSE  SYSTEMS 
MANAGEMENT  COLLEGE 


FOREWORD 


This  book  is  one  of  a  family  of  educational  guides  written 

from  a  Department  of  Defense  perspective;  i.e.,  non-Service  peculiar. 
They  are  intended  primarily  for  use  in  the  courses  at  the  Defense  Systems 
Management  College  (DSMC)  Technical  Management  Department  and  secondarily 
as  a  desk  reference  for  program  and  project  management  personnel.  These 

books  are  written  for  current  and  potential  Department  of  Defense  (DOD) 

acquisition  management  personnel,  who  have  some  familiarity  with  the 
basic  terms  and  definitions  employed  in  program  offices.  They  are  designed 
to  assist  both  government  and  industry  personnel  in  executing  their 

management  responsibilities  relative  to  the  acquisition  and  support  of 
defense  systems.  They  include: 

a.  Integrated  Logistic  Support  (ILS)  Guide;  First  Edition;  May  1986 

b.  Embedded  Computer  Resources  (ECR)  Guide;  estimated  publication 
date:  1988 

c.  Systems  Engineering  Management  Guide;  December  1986 

d.  Department  of  Defense  Manufacturing  Management  Handbook  ^or 
Program  Managers;  Second  Edition;  July  1984. 

This  family  of  books  is  especially  needed  at  this  time.  The 

Government  desires  capable,  producible,  supportable,  testable  systems 
to  be  brought  in  within  cost  and  schedule.  The  increasing  cost  and 
technical  complexity  of  defense  systems,  however,  has  forced  greater 
specialization  of  functions  and  the  rise  of  many  specific  (and  very  often 
vocal)  disciplines.  Public  attention  to  the  Defense  Acquisition  Process 
has  also  intensified.  The  key  to  a  successful  program  is  intelligent 
integration  and  balance  among  the  many  disciplines  that  constitute  a 
modern  system.  This  is  achieved  through  a  process  that  begins  with 

communication  among  different  disciplines  and  continues  with  a  careful 
tradeoff  process  throughout  the  system  life  cycle. 

Each  of  the  books  has  a  common  foreword  designed  to  assist 
managers  in  sharpening  their  judgement  and  focusing  their  thinking. 
They  are  not  to  be  used  as  an  all  inclusive  checklist  or  model  of  the 

single  correct  approach  to  system  acquisition  management,  because  all 
programs  are  unique  and  must  be  executed  with  professional  judgement 
and  common  sense. 

This  book  was  developed  by  THE  BDM  CORPORATION  under  contract 
MDA  903-87-C-0046,  directed  by  DSMC.  Special  thanks  are  due  members  of 
the  The  BDM  Corporation  staff,  DSMC  faculty,  students,  alumni,  and  members 
of  the  acquisition  community  at  large,  whose  comments,  suggestions,  and 

materials  were  helpful  in  completing  this  project.  The  Defense  Systems 
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Management  College  Is  the  controlling  agency  for  this  book.  Comments 
and  recommendations  relating  to  the  text  are  solicited.  You  are  encouraged 
to  place  them  on  one  of  the  pre-addressed  tear  sheets  located  at  the 
back  of  the  book  and  mail  them  to  us. 

This  introduction  offers  a  systems  perspective  for  the  technical 
management  of  a  specific  discipline  during  the  system  life  cycle. 
Subsequent  material  in  this  book  provides  a  guide  for  managing  specific 
discipline  within  this  broad  scope  of  technical  activities.  The  past 
several  decades  have  seen  the  rise  of  large,  highly  interactive  defense 
systems  that  are  often  on  the  forward  edge  of  technology.  These  systems 
have  a  natural  process  of  evolution,  or  life  cycle,  in  which  actions 
taken  or  avoided  In  the  very  early  stages  can  mean  the  difference  between 
success  and  failure  downstream. 

The  system  life  cycle  consists  of  the  interval  from  program 
initiation  to  system  disposal.  All  activity  in  the  acquisition  process 
centers  around  the  system/equipment.  Thus,  the  state  of  definition  of 
the  system  configuration  at  any  time  in  the  system  life  cycle  is  an  area 
of  common  Interest  among  all  disciplines.  Phases  in  the  defense  system's 
life  cycle  are  concept  exploration/definition  concept, 
demonstration/validation,  full  scale  development,  full-rate 
production/deployment,  and  operational  and  support. 

The  division  of  technical  activities  Into  functional  areas 
of  design,  test,  manufacturing,  and  logistic  support  is  convenient  and 
usually  results  In  a  corresponding  division  of  labor  in  a  program  office. 
As  can  be  seen  from  Figure  1-1,  each  of  these  functional  areas  is  active 
in  the  earliest  phase  of  the  life  cycle  and  continues  through  most  of 
the  program.  The  general  thrust  of  technical  management  is  : 

Define  what  It  takes  to  support,  produce,  and  test  the  system 
utilizing  analyses.  Then  see  if  we  can  afford  it. 

Influence  the  design  through  producibility  engineering,  logistics 
analysis,  testability  design,  and  design  to  cost.  Develop 
specifications  and  translate  y,equ1rements  to  contract  language. 
Prepare  to  execute  by  cuf urging  for  the  test  facilities, 
acquiring  and  setting  up  production  line,  and  designing 

and  acquiring  the  logistic  su^ort. 

Execute  by  testing,  manufacturing,  supporting. 

Figure  F-l  Is  a  rigorous  endeavor  to  show  technical  management 
activities  In  relative  time  phase.  As  such,  it  identifies  activities 
that  should  be  accomplished  and  Integrated  In  the  various  program  phases. 

Acquisition  of  a  system  is  a  process  that  begins  with  the 
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Identification  of  a  need.  The  goal  of  a  system  acquisition  Is  to  deploy 
(in  a  timely  manner)  and  sustain  an  effective  system  that  satisfies  the 
need  at  an  affordable  cost. 

Thus,  the  effort  involved  in  the  acquisition  process  can  be 
modeled  as  an  input,  process,  and  output.  The  output  is  the  system. 
The  input  is  the  need  and  other  appropriate  constraints.  The  process 
consists  of  managing  the  technical  activities  by  establishing  and 
maintaining  a  balance  among  cost  (the  resources  required  to  acquire, 
produce,  operate  and  support,  and  dispose  of  a  system),  system 
effectiveness  (the  degree  to  which  a  system  can  be  expected  to  achieve 
a  set  of  specific  mission  requirements),  and  schedule.  Much  of  the 
criticism  leveled  at  the  00D  results  from  a  perception  of  imbalance  among 
these  factors. 


POST  SCRIPT 

The  test  and  evaluation  process  for  DoD  materiel  acquisitions 
is  a  complex  exercise  of  integrating  the  data  collection  necessary  to 
satisfy  the  Program  Manager's  information  requirements  on  system 
performance.  When  poorly  managed,  test  and  evaluation  events  can  generate 
schedule  slips  or  adverse  media  exposure  leading  to  intensive  OSD  and/or 
Congressional  Interest  in  a  program's  status. 

The  objective  of  a  well  managed  test  and  evaluation  program 
Is  to  provide  the  Program  Manager  with  timely  and  accurate  information 
to  support  his  decision  making  process.  This  guide  has  been  developed 
to  assist  the  acquisition  community  obtain  a  better  understanding  of 
the  complexities  of  test  and  evaluation. 


John  Claxton 
Course  Director 
Test  and  Evaluation  Management 
Course 
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MODULE  I 


INTRODUCTION  TO  TEST  AND  EVALUATION 


This  first  module  introduces  the  test  and  evaluation  process, 
describes  the  relationship  of  T&E  to  the  defense  systems  acquisition 
cycle,  and  discusses  the  role  of  T&E  in  the  overall  systems  engineering 
process.  The  module  also  defines  the  different  types  of  T&E  used  in 
defense  system  acquisition  programs  and  introduces  frequently  used  test 
and  evaluation  documents.  The  module  concludes  with  a  brief  discussion 
of  the  U3e  of  nondevelopment  items  in  defense  system  acquisition  programs 
and  explores  the  implications  of  this  use  on  test  and  evaluation 
activities . 


CHAPTER  1 

THE  TEST  AND  EVALUATION  PROCESS 


1.1  INTRODUCTION 


The  fundamental  purpose  of  test  and  evaluation  in  a  defense 
system's  development  and  acquisition  program  is  to  identify  the  areas 
of  risk  to  be  reduced  or  eliminated.  During  the  early  phases  of 
development,  T&E  is  conducted  to  demonstrate  the  feasibility  of  conceptual 
approaches,  to  minimize  design  risk,  to  identify  design  alternatives, 
to  compare  and  analyze  tradeoffs,  and  to  estimate  operational  effectiveness 
and  suitability.  As  a  system  undergoes  design  and  development,  the 
emphasis  in  testing  moves  gradually  from  development  test  and  evaluation 
(DT&E),  which  is  concerned  chiefly  with  the  attainment  of  engineering 
design  goals,  to  operational  test  and  evaluation  (OffcE),  which  focuses 
on  questions  of  operational  effectiveness,  suitability,  Jmd  supportability. 
Although  there  are  usually  clearly  separate  development  and  operational^ 
test  events,  DT&E  and  OT&E  are  not  necessarily  serial  phases  in  the 
evolution  of  a  weapon  system.  In  fact,  combined  and  concurrent  development 
and  operational  testing  is  encouraged  when  appropriate  (Reference  16). 


T&E  has  its  origins  in  the  testing  of  hardware;  this  tradition 
is  heavily  embedded  in  its  vocabulary  and  procedures.  The  advent  of 
software  intensive  systems  has  brought  with  it  new  challenges  and  new 
approaches  to  testing  that  are  discussed  in  Chapter  12  of  this  management 
guide.  What  remains  constant  throughout  the  T&E  process,  whether  testing 
hardware  or  software,  is  the  need  for  thorough,  logical,  systematic, 
and  early  test  planning  and  the  feedback  of  well  documented,  unbiased 
test  and  evaluation  results  to  system  developers,  users,  and 
decisionmakers.  ’V 

I 


Test  j and  evaluation  serves  a  number  of  useful  functions, 
providing  info/mation  for  a  variety  of  costumers.  T&E  provides  information 
to  developers (to  assist  in  the  identification  and  resolution  of  technical 
difficulties.  ^T&E  provides  information  to  decision  makers  responsible 
for  making  the  investment  decision  to  procure  a  new  system  and  for  deciding 
on  the  most  effective  use  of  limited  resources.  Moreover,  T&E  provides 
information  to  operational  users  to  support  the  development  of  effective 
tactics,  doctrine,  and  procedures.^  Figure  1-1  highlights  some  of  the 
users  of  T&E  information.  \ 
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DEFENSE  SYSTEM  ACQUISITION  PROCESS 


The  defense  system  acquisition  process  underwent  revision  in 
1987  in  an  attempt  to  make  it  less  costly,  less  time  consuming,  and  more 
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Figure  1-1.  Users  of  THE  Information 


responsive  to  the  needs  of  the  operational  community.  As  it  is  now 
structured,  the  defense  system  life  cycle  consists  of  the  following  five 
phases: 

(1)  Concept  Exploration/Definition 

(2)  Concept  Demonstration/Validation 

(3)  Full-Scale  Development 

(4)  Full-Rate  Production/Deployment 

(5)  Operational  Support 

As  Figure  1-2  shows,  these  phases  are  separated  by  key  decision  points 
(milestones)  when  a  decision  authority  reviews  a  program  and  authorizes 
It  to  advance  to  the  next  stage  in  the  cycle.  T&E  results  and  planned 
T&E  play  an  Important  part  and  are  rigorously  assessed  as  part  of  the 
milestone  review  process. 

The  following  brief  description  of  the  defense  system  acquisition 
process  is  provided  to  permit  the  reader  to  place  test  and  evaluation 
within  the  context  of  the  larger  process.  The  description  is  based 
primarily  upon  Information  found  in  DoD  Instruction  5000.2  (Reference 
16). 


1.2.1  Concept  Exploration/Definition  Phase 

The  defense  system  acquisition  process  begins  with  the  submission 
of  a  Mission  Need  Statement  with  the  Service's  Program  Objective  Memorandum 
(POM).  Secretary  of  Defense  (SECDEF)  approval  becomes  the  program 
Initiation  decision  (Milestone  0).  A  Concept  Exploration  Phase  follows 
during  which  alternative  approaches  for  satisfying  the  requirement  usually 
In  the  form  of  breadboard  configurations  are  investigated.  The  Concept 
Exploration  Phase  concludes  with  the  Milestone  I  selection  of  a  concept 
or  concepts  to  enter  a  Concept  Demonstration/Validation  Phase.  The 
Milestone  I  decision  establishes  broad  thresholds  for  program  cost, 
schedule,  and  operational  effectiveness  and  suitability.  Key  documents 
for  the  T&E  manager  at  the  time  of  the  Milestone  I  review  are  the  System 
Concept  Paper  (SCP),  the  Test  and  Evaluation  Master  Plan  (TEMP),  and 
the  Integrated  Logistics  Support  Plan  ( ILSP)/Logistics  Support  Analysis 
(LSA).  Additional  program  management  documents  prepared  prior  to  Milestone 
I  Include:  the  Competitive  Prototyping  Strategy  Document  (if  required), 
the  Cost  and  Operational  Effectiveness  Analysis  (COEA)  Report,  the 
Common-Use  Alternatives  Statement,  Independent  Cost  Estimates,  and  the 
Program  Baseline  that  summarizes  the  weapon's  functional  specifications, 
performance  parameters,  and  cost  and  schedule  objectives. 

1.2.2  Concept  Deaonstratlon/Valldatlon  Phase 

After  the  Milestone  I  decision,  the  program  enters  the  Concept 
Demonstration/Validation  Phase  during  which  selected  concepts,  typically 
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brassboard  or  early  prototype,  are  refined  through  study  and  analysis. 
This  phase  ends  with  the  Milestone  II  decision  to  either  enter  into 
full-scale  development  or  terminate  the  program.  The  Milestone  II  decision 
establishes  more  specific  cost,  schedule,  and  operational  effectiveness 
and  suitability  goals  and  thresholds.  Documents  of  particular  interest 
to  the  T&E  manager  at  the  time  of  the  Milestone  II  review  include  the 
Decision  Coordinating  Paper  (DCP)  and  the  updated  TEMP.  Additional 
documents  prepared  or  updated  at  this  time  include:  the  COEA  report, 
the  Common  Use  Alternatives  Statement,  Independent  Cost  Estimates,  the 
Manpower  Estimate  Report,  the  Acquisition  Strategy  Report,  the  Development 
Baseline,  and  the  Operational  Assessment  for  LRIP  Funding  or  Long  Lead 
Production  Items. 

1.2.3  Full-Scale  Development  Phase 

During  the  Full-Scale  Development  Phase,  the  selected  system 
and  its  principal  items  of  support  are  fabricated.  This  phase  ends  with 
the  Milestone  III  decision  to  produce  the  system.  Key  documents  for 
the  T&E  manager  at  the  time  of  the  Milestone  III  review  are  an  updated 
DCP,  an  updated  TEMP,  and  the  Beyond-Low-Rate  Initial  Production  Report. 
The  Beyond  LRIP  Report  is  required  by  law  of  the  Director  of  Operational 
T&E  to  document  his  assessment  of  the  adequacy  of  operational  test  and 
evaluation  and  the  reported  operational  effectiveness  and  suitability 
of  the  system.  Also  mandated  by  law  is  the  requirement  to  conduct  Live 
Fire  testing  to  proceed  beyond  LRIP.  Other  reports  prepared  in  conjunction 
with  Milestone  III  include  Independent  Cost  Estimates,  the  Manpower 
Estimate  Report,  the  Acquisition  Strategy  Report,  and  the  Production 
Baseline.  The  results  of  completed  OT&E  are  carefully  considered  at 
the  time  of  the  Milestone  III  decision. 

1.2.4  Full-Rate  Production  and  Operational  Support  Phases 

The  Milestone  III  decision  is  followed  by  a  Full-Rate 
Production/Deployment  Phase.  This  phase  ends  with  a  Logistics  Readiness 
and  Support  Review  (Milestone  IV)  to  identify  the  actions  and  resources 
needed  to  achieve  and  maintain  operational  readiness  and  support  objectives 
for  the  first  several  years  of  the  fifth  and  final  phase,  the  Operational 
Support  Phase.  The  Milestone  V  decision  at  the  end  of  the  Operational 
Support  Phase  encompasses  a  review  of  a  system's  operational  effectiveness, 
suitability,  and  readiness  to  determine  whether  major  upgrades  are 
necessary  or  deficiencies  warrant  consideration  of  replacement.  In 
preparation  for  Milestones  IV  and  V,  the  DCP,  the  TEMP,  and  the  Production 
Baseline  are  updated  to  describe  the  program  status,  changes,  and  issues. 
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1.3  T4E  and  the  Systems  Engineering  Process 

In  the  early  1970s,  Department  of  Defense  (DoD)  test  policy 
became  more  formalized  and  placed  greater  emphasis  on  test  and  evaluation 
(T&E )  as  a  continuing  function  throughout  the  acquisition  cycle.  These 
policies  stressed  the  use  of  T&E  to  reduce  acquisition  risk  and  provide 
early  and  continuing  estimates  of  the  system's  operational  effectiveness 
and  operational  suitability.  To  meet  these  objectives,  appropriate  test 
activities  had  to  be  fully  integrated  into  the  overall  development  process. 
From  a  systems  engineering  perspective,  test  planning,  testing,  and 
analysis  of  test  results  are  integral  parts  of  the  basic  product  definition 
process. 

MIL-STD-499  defined  systems  engineering  in  the  DoD  context: 

System  Engineering  is  the  application  of  scientific 
and  engineering  efforts  to  (a)  transform  an  operational 
need  into  a  description  of  system  performance  parameters 
and  a  system  configuration  through  the  use  of  an 
iterative  process  of  definition,  synthesis,  analysis, 
design,  test,  and  evaluation;  (b)  integrate  related 
technical  parameters  and  assure  compatabil ity  of  all 
physical,  functional,  and  program  interfaces  in  a  manner 
that  optimizes  the  total  system  definition  and  design; 

(c)  integrate  reliability,  maintainabil ity,  safety, 
survivability,  human,  and  other  such  factors  into  the 
total  engineering  effort  to  meet  cost,  schedule,  and 
technical  performance  objectives. 

A  system's  life  cycle  begins  with  the  user's  needs,  which  are 
expressed  as  constraints,  and  the  capability  requirements  needed  to  satisfy 
mission  objectives.  Systems  engineering  is  essential  in  the  earliest 
planning  period,  in  conceiving  the  system  concept,  and  defining  performance 
requirements  for  system  elements.  As  the  detailed  design  is  being  done, 
systems  engineers  ensure  balanced  influence  of  all  required  design 
specialties  (including  "testability"),  resolve  interface  problems,  perform 
design  reviews,  perform  trade-off  analyses,  and  assist  in  verifying 
performance. 

The  days  in  which  any  one  or  two  individuals  can  design  a  complex 
system,  especially  one  of  the  huge  weapon  systems  of  the  modern  age,  are 
past.  The  systems  are  too  complex  and  require  too  much  indepth  knowledge 
over  too  broad  a  range  of  areas  and  technical  disciplines  for  a  small 
number  of  generalists  to  accommodate.  System  engineers  coordinate  the 
many  specialized  design  engineers  and  are  responsible  for  the  integration 
of  the  pieces  into  a  system. 

Systems  engineering  through  interdisciplinary  integration  manages 
the  progress  of  product  definition  from  system  level,  to  configuration 
item  level,  detailed  level,  deficiency  correction,  and  modifications/product 
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improvements.  Test  results  provide  feedback  for  analysis  of  design  progress 
toward  performance  goals.  The  tools  of  systems  engineering  include  design 
reviews,  configuration  management,  simulation,  technical  performance 
measurement,  trade-off  analysis,  and  specifications. 

What  products  does  systems  engineering  produce?  It  gives  answers 
to  what  specialists  are  required,  what  segments  to  use,  what 
nondevelopmental  items  to  use,  design  performance  limits  and  trade-off 
criteria,  how  to  test,  when  to  test,  how  to  document  (specifications), 
and  what  management  controls  to  apply  (technical  performance  measurement 
and  design  reviews). 

Figure  1-3  depicts  development  testing  (DT)  and  operational 
testing  (OT)  support  of  the  technical  reviews  used  to  monitor  the  systems 
engineering  process.  More  information  on  the  reviews  is  contained  in 
Chapter  8. 

1,3.1  The  Systems  Engineering  Process 

The  systems  engineering  process  is  the  iterative  logical  sequence 
of  analysis,  design,  test  and  decision  activities  that  transforms  an 
operational  need  into  the  descriptions  required  for  production  and  fielding 
of  all  operational  and  support  system  elements.  This  process  consists 
of  four  activities.  They  include  functional  analysis ,  synthesis,  evaluation 
and  decision  (trade-off),  and  description  of  system  elements. 

The  functional  analysis  activity  identifies  “what"  the  system, 
component,  or  part  must  do.  It  works  from  the  top,  downward  assuring 
requirements  traceability.  It  reveals  alternative  concepts.  This  is 
done  without  assuming  "how"  functions  will  be  accomplished.  The  product 
is  a  series  of  alternative  Functional  Flow  Block  Diagrams  (FFBD).  A 
functional  analysis  can  be  applied  at  every  level  of  development.  At 
the  system  level,  it  may  be  a  contractor  or  Service  effort.  Developmental 
testers  assist  the  functional  analysis  activity  during  the  concept 
exploration  phase  to  help  determine  "what"  each  component's  role  will 
be  as  a  part  of  the  system  being  developed. 

The  synthesis  activity  involves  invention;  conceiving  ways  to 
do  each  FFBD  task;  to  answer  the  "how"  question.  Next,  the  physical 
interfaces,  which  the  how  answers  imply,  are  carefully  identified 
(topological  or  temporal).  All  answers  must  reflect  all  technology 
selection  factors.  Synthesis  tools  include  Requirements  Allocation  Sheets 
(RAS)  which  translate  functional  statements  into  design  requirements, 
permitting  a  complex,  long,  interactive  invention  process  with  control, 
visibility,  and  requirements  traceability.  Developmental  testers  conduct 
Demonstration/Validation  testing  to  determine  how  the  components  will 
perform  their  assigned  functions  to  assist  this  synthesis  activity. 
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SYSTEMS  ENGINEERING 
MANAGEMENT 


Defense  Systems  Management  College 


The  evaluation  and  decision  activity  is  a  trade-off  of  alternative 
approaches  to  "how".  This  activity  is  conducted  in  accordance  with  decision 
criteria  set  by  higher  level  technical  requirements  for  such  things  as 
life  cycle  costs,  effectiveness,  reliability,  availability,  maintainability, 
risk  limits,  schedule,  etc.  It  is  repeated  at  each  level  of  development. 
The  evaluation  and  decision  activity  is  assisted  by  developmental  testers 
during  the  later  Demonstration/Validation  phase  and  the  Full-Scale 
Development  phase  when  competitive  testing  between  alternative  approaches 
Is  performed. 

The  final  activity  Is  a  description  of  system  elements.  This 
occurs  as  the  results  of  the  previous  activities,  as  the  final  system's 
design  is  determined.  This  takes  form  as  the  specifications  that  are 
verified  in  testing  and  reviewed  in  the  Physical  Configuration  Audit  and 
the  Functional  Configuration  Audit.  Operational  testers,  during  the 
Full-Scale  Development  phase,  assist  in  this  activity.  They  conduct 
operational  testing  of  the  test  items/systems  to  help  determine  the 
personnel,  equipment,  facilities,  software,  and  technical  data  requirements 
of  the  new  system  when  used  by  typical  military  personnel.  Figure  1-4, 
System  Engineering  Process,  depicts  the  four  activities  that  take  place 
and  their  interaction. 

1.3.2  The  System  Engineering  Management  Plan 

The  System  Engineering  Management  Plan  (SEMP)  is  a  concise  top 
level  management  plan  for  the  integration  of  all  system  design  activities. 
Its  purpose  is  to  make  visible  the  organization,  direction  and  control 
mechanisms,  and  personnel  for  the  attainment  of  cost,  performance,  and 
schedule  objectives.  The  SEMP  defines  and  describes  the  type  and  degree 
of  system  engineering  management,  the  system  engineering  process,  and 
the  integration  of  related  engineering  programs.  The  design  evolution 
process  described  in  the  SEMP  forms  the  basis  for  comprehensive  test  and 
evaluation  planning. 

The  TEMP  must  be  consistent  with  the  SEMP.  The  testing  program 
outlined  in  the  TEMP  must  provide  the  technical  performance  measurements 
data  required  for  all  design  decision  points,  audits,  and  reviews  that 
are  a  part  of  the  system's  engineering  process  outlined  in  the  SEMP. 
The  configuration  management  process  outlined  in  the  SEMP  controls  the 
baselines  for  the  test  programs  and  incorporates  design  modifications 
to  the  baseline  determined  to  be  necessary  by  T&E. 

The  TEMP  and  the  SEMP  must  be  traceable  to  each  other.  The 
system  description  in  the  TEMP  must  be  traceable  to  systems  engineering 
documentation  such  as  the  Functional  Flow  Block  Diagrams  (FFBDs),  the 
Requirements  Allocation  Sheets  (RASs),  and  the  Test  Requirements  Sheets 
(TRSs).  Key  functions  and  interfaces  of  the  system  with  other  systems 
must  be  described  and  correlated  with  the  systems  engineering  documentation 
and  the  system  specification  (Type  A).  Operational  and  technical  thresholds 
in  the  SEMP  include  specific  performance  requirements  which  become  test 
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planning  limits.  They  must  be  traceable  through  the  planned  systems 
engineering  documentation  and  can  be  coorelated  to  the  content  of  the 
TPM  program.  Failure  criteria  for  reliability  thresholds  during  OT&E 
testing  must  be  delineated  and  agreed  upon  by  the  Program  Manager  and 
the  Operational  Test  Director  and  reflected  in  the  SEMP  and  the  TEMP. 

1.3.3  Technical  Performance  Measurement 

The  concept  of  technical  performance  measurement  (TPM)  is  one 
of  identifying  critical  technical  parameters  which  are  at  risk  during 
design,  then  tracking  evaluation  and  test  data;  making  predictions  about 
whether  or  not  the  parameter  can  achieve  final  technical  success  (within 
the  allocated  resources),  and  then  using  these  data  to  assist  in  managing 
the  technical  program. 

The  technical  performance  measurement  program  is  an  Integral 
part  of  the  T&E  program.  TPM  is  defined  as  product  design  assessment 
and  forms  the  backbone  of  the  development  testing  program.  It  estimates, 
through  engineering  analyses  and  tests,  the  values  of  essential  performance 
parameters  on  the  current  design  in  a  program.  It  serves  as  a  major  input 
in  the  continuous  overall  evaluation  of  operational  effectiveness  and 
suitability.  Design  reviews  are  conducted  to  measure  the  progress  of 
the  systems  engineering  progress.  For  more  information,  see  Chapter  8. 
Figure  1-5  depicts  the  technical  reviews  that  usually  take  place  during 
the  systems  engineering  process  and  the  related  specification  documents. 

1.3.4  Product  Baselining  and  T&E 

The  systems  engineering  process  establishes  a  product  baseline 
throughout  the  acquisition  cycle.  This  baseline  can  be  modified  with 
the  results  of  engineering  and  testing.  The  testing  to  prove  the  technical 
or  development  baseline  rarely  is  the  same  as  the  baseline  for  the 
operational  testing  or  the  production  baseline. 

Related  to  the  product  baseline  is  the  process  of  configuration 
management.  Configuration  management  benefits  the  test  and  evaluation 
community  in  two  ways.  Through  configuration  management,  the  baseline 
product  to  be  used  for  testing  is  determined.  Also,  changes  that  occur 
to  the  baseline  as  a  result  of  testing  and  design  reviews  are  incorporated 
into  the  test  article  prior  to  the  new  phase  of  testing  (to  prevent  retest 
of  a  bad  design). 

1.4  DEFINITIONS 

T&E  is  the  deliberate  and  rational  generation  of  data  concerning 
the  nature  of  the  emerging  system  and  the  creation  of  information  useful 
to  the  technical  and  managerial  personnel  controlling  its  development. 
In  the  broad  sense,  T&E  may  be  defined  as  all  physical  testing,  modeling. 
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simulation,  and  experimentation  and  related  analyses  performed  during 
the  course  of  the  research,  development,  introduction  and  employment  of 
a  weapon  system  or  subsystem.  The  Glossary:  Defense  Acquisition  Acronyms 
and  Terms,  Defense  Systems  Management  College,  July  1987,  defines  "Test" 
and  "Test  and  Evaluation"  as  follows: 

A  "test"  is  any  program  or  procedure  which  is  designed  to  obtain, 
verify,  or  provide  data  for  the  evaluation  of:  research  and  development 
(other  than  laboratory  experiments);  progress  In  accomplishing  development 
objectives;  or  performance  and  operational  capability  of  systems, 
subsystems,  components,  and  equipment  items. 

"Test  and  Evaluation"  is  the  process  by  which  a  system  or 
components  are  compared  against  requirements  and  specifications  through 
testing.  The  results  are  evaluated  to  assess  progress  of  design, 
performance,  supportability,  etc. 

1.5  SUftfARY 


Test  and  evaluation  is  a  technical  management  tool  used  to  reduce 
risk  throughout  the  defense  system  acquisition  cycle.  This  cycle  consists 
of  five  phases  separated  by  discrete  milestones.  T4E  results  are  used 
to  support  the  design  reviews  that  form  an  important  part  of  the  system 
engineering  process  used  by  system  developers  and  to  aid  in  the  milestone 
decision  process  used  by  senior  decision  authorities  in  the  Department 
of  Defense. 
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CHAPTER  2 

IMPORTANCE  OF  TEST  AND  EVALUATION 


2.1  INTRODUCTION 

Risk  analysis  is  the  process  of  determining  the  probability 
that  a  specific  combination  of  technical  performance,  schedule,  and  cost 
will  or  will  not  be  attained  if  a  planned  course  of  action  Is  followed. 
The  responsibilities  of  a  Program  Manager  and  decision  authorities  center 
on  assessing  and  controlling  risk.  This  chapter  describes  how  test  and 
evaluation  functions  as  a  risk  management  tool.  It  also  addresses  the 
contribution  TAE  makes  through  the  provision  of  empirical  data  before 
each  milestone  review.  The  support  TAE  provides  to  the  design  review 
process  is  addressed  in  Chapter  8. 

2.2  TESTING  AS  A  RISK  MANAGEMENT  TOOL 

As  shown  in  Figure  2-1,  the  cost  of  correcting  defects  in  weapons 
has  been  estimated  to  add  from  10  to  30  percent  to  the  cost  of  each  item 
(Reference  107).  Such  costly  redesign  and  modification  efforts  can  be 
reduced  if  carefully  planned  and  executed  test  and  evaluation  programs 

are  used  to  detect  and  fix  system  deficiencies  sufficiently  early  in 

the  acquisition  process. 

In  1983,  the  Assistant  Secretary  of  Defense  made  the  following 
statement  regarding  the  Importance  of  TAE  to  the  Senate  Committee  on 
Governmental  Affairs: 

.  .  .  the  criterion  should  not  be  how  quickly  we 
can  field  any  new  weapon,  but  rather  how  quickly 

we  can  field  a  new  weapon  that  works.  The  only  weapons 
that  would  be  significantly  delayed  would  be  the 

ones  that  operational  testing  shows  to  be  unsuitable 
for  combat,  and  I  cannot  believe  that  any  of  us  would 
advocate  saddling  our  fighting  forces  with  any  of 
those.  In  fact,  the  most  likely  effect  of  operational 
testing  Is  not  to  delay,  but  to  accelerate  the 
development  process.  Trying  to  fix  a  faulty  weapon 
after  it's  in  the  field  --  if  It  can  still  be  fixed 
—  Is  a  far  slower  process  than  fixing  the  design 
before  it  goes  into  production. 
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Figure  2-1.  Estimated  Cost  Increase  to  Correct  Defects 


Thus,  test  and  evaluation  may  reduce  cost,  schedule  and  technical  risks. 
There  is  a  third  type  of  risk  involved  in  the  development  and  acquisition 
of  new  systems  -technical  risk.  Test  and  evaluation  of  parts,  components, 
subsystems,  and  systems  can  also  be  used  to  estimate  and  manage  this 
technical  risk. 

Test  and  evaluation  results  figure  prominently  in  the  decisions 
reached  at  design  and  milestone  reviews.  However,  the  fact  that  T&E 
results  are  required  at  major  decision  points  does  not  presuppose  that 
T&E  results  must  always  be  favorable.  The  final  decision  responsibility 
lies  with  the  executive  who  must  examine  the  critical  Issues  and  weigh 
the  facts  at  hand.  Only  he  can  determine  the  weight  and  importance 
that  is  to  be  attributed  to  a  system's  diverse  capabilities  and 
shortcomings  —  only  he  can  decide  the  degree  of  risk  he  is  willing  to 
accept.  The  decision  authority  will  be  unable  to  make  this  decision 
without  a  solid  base  of  information  provided  by  test  and  evaluation. 
Figure  2-2  illustrates  the  Life  Cycle  Cost  of  the  System  and  how  decisions 
impact  the  expenditures  on  the  program. 

A  Defense  Science  Board  1983  Task  Force  focused  on  the  reduction 
of  risk  in  program  acquisition  (Reference  42).  This  group  made  the 
following  observations: 

o  A  poorly  designed  product  cannot  be  properly  tested  or  produced. 

o  The  control  techniques  needed  to  successfully  complete  the 

design,  test,  and  production  of  an  item  dictate  the  management 
system  required. 

o  The  industrial  process  of  weapon  system  acquisition  demands 

a  better  understanding  and  Implementation  of  basic  engineering 
and  manufacturing  disciplines. 

o  The  industrial  process  is  focused  on  the  design,  test,  and 

production  of  a  product. 

o  The  design,  test,  and  production  processes  are  a  continuum 

of  Interdependent  disciplines.  A  failure  to  perform  well  in 
one  area  will  result  in  failure  to  do  well  in  all  areas.  When 
this  happens  --  as  It  does  all  too  often  --  a  high-risk  program 
results  whose  equipment  is  fielded  later  and  at  far  greater 
cost  than  planned. 

The  Task  Force  developed  a  set  of  templates  for  use  in  establishing  and 
maintaining  low-risk  programs.  Each  template  describes  an  area  of  risk 
and  then  specifies  technical  methods  for  reducing  that  risk.  Program 
Managers  and  test  managers  may  wish  to  consult  these  templates  for  the 
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M7.M.S719.UM2  Figure  2-2  Life-Cycle-Cost  Decision  Impact  and  Expenditures 

Source:  Defense  Systems  Management  College 


guidance  they  provide  in  reducing  the  risks  frequently  associated  with 
test  programs  (Reference  42,  pages  4-10,  4-11).  A  sample  risk  management 
template  is  shown  in  Figure  2-3. 

2.3  THE  T&E  CONTRIBUTION  AT  MAJOR  MILESTONES 

Test  and  evaluation  progress  is  monitored  by  the  Office  of 
the  Secretary  of  Defense  (OSD)  throughout  the  acquisition  process.  Their 
oversight  extends  to  the  major  materiel  acquisitions  or  designated 
acquisitions  which  is  about  5%  of  all  the  acquisitions  being  managed 
within  DoD.  T&E  officials  within  OSD  render  independent  assessments 
to  the  Defense  Acquisition  Board,  the  Defense  Acquisition  Executive, 
and  the  Secretary  of  Defense  at  each  major  system  milestone.  These 
assessments  are  based  on  the  following  T&E  information: 

o  The  Test  and  Evaluation  Master  Plan  (TEMP)  and  more  detailed 
supporting  documents  developed  by  responsible  Service 
activities. 

o  Service  test  agency  reports  and  briefings;  and, 

o  Development  test  and  evaluation  data  from  sources  such 

as  the  Service  program  managers,  laboratories,  industry 
developers,  and  studies  and  analyses. 

At  Milestone  I,  the  OSD  T&E  assessment  reflects  an  evaluation 
of  system  concepts  and  alternatives  based  on  specific  goals  and  thresholds 
established  in  an  approved  TEMP.  At  Milestone  II,  it  includes  an 
assessment  of  previously  established  test  plans  and  test  results.  At 
Milestone  III,  reviews  verify  the  operational  effectiveness  and  suitability 
of  major  weapon  systems. 

A  primary  contribution  made  by  T&E  is  the  detection  and  reporting 
of  deficiencies  that  may  adversely  impact  the  performance  capability 
or  availability/supportability  of  a  system.  A  deficiency  reporting  process 
is  used  throughout  the  acquisition  process  to  report,  evaluate,  and  track 
system  deficiencies  and  to  provide  the  impetus  for  corrective  actions. 


2.3.1  Test  Contributions  Prior  to  Milestone  I 


During  the  Concept  Exploration/Definition  Phase  prior  to 
Milestone  I,  laboratory  testing,  modeling  and  simulations  are  conducted 
by  the  contractor  and  the  Development  Agency  to  demonstrate  and  assess 
the  capabilities  of  key  subsystems  and  components.  The  test  and  simulation 
designs  are  based  on  the  requirements  documented  in  the  Mission  Need 
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SOURCE:  'SOLVING  THE  RISK  EQUATION  IN  TRANSITIONING  FROM  DEVELOPMENT  TO  PRODUCTION, 
DEFENSE  SCIENCE  BOARD  TASK  FORCE  REPORT,  25  MAY  1983 


Statement.  Studies,  analyses,  simulation,  and  test  data  are  used  by 
the  Development  Agency  to  explore  and  evaluate  alternative  concept  designs 
proposed  to  satisfy  the  requirements.  Also  during  this  period,  the 
Operational  Test  and  Evaluation  Agency  (OTA)  monitors  concept  exploration 
T&E  to  gather  information  for  future  test  and  evaluation  planning  and 
to  provide  effectiveness  and  suitability  inputs  desired  by  the  Program 
Manager.  The  OTA  also  conducts  operational  assessments,  as  feasible, 
to  assess  the  operational  impact  of  candidate  technical  approaches  and 
to  assist  in  selecting  preferred  alternative  systems  concepts. 

Toward  the  end  of  the  phase,  the  Development  Agency  prepares 
the  DT&E  System  Concept  Report  to  record  and  present  T&E  results  of 
engineering  and  performance  evaluations  of  system  design(s)  compared 
to  stated  requirements  and  concept  specifications.  The  information  in 
this  report  is  incorporated  into  the  Program  Manager's  Status  Briefing 
and  the  System  Concept  Paper,  key  documents  that  form  the  basis  for  the 
Milestone  I  decision  to  proceed  to  the  Concept  Demonstration  and  Validation 
Phase. 

2.3.2  Test  Contributions  Prior  to  Milestone  II 

During  the  Concept  Demonstration/Validation  Phase  prior  to 
Milestone  II,  concepts  approved  for  demonstration  and  validation  form 
the  baseline  which  are  used  for  detailed  test  planning. 

The  Development  Agency  conducts  development  test  and  evaluation 
during  the  Demonstration/Validation  Phase  to  assist  with  engineering 
design,  system  development,  and  to  verify  attainment  of  technical 
performance  specifications,  and  program  objectives.  DT&E  includes  T&E 
of  components,  subsystems,  and  prototype  development  models.  T&E  of 
functional  compatibility  and  interoperability  with  existing  and  planned 
equipment  and  systems  is  also  included.  During  this  phase  of  testing, 
adequate  DT&E  is  accomplished  to  ensure  that  engineering  is  reasonably 
complete  (including  survivability/vulnerability,  compatibility, 
transportability,  interoperabil ity,  reliability,  maintainability,  safety, 
human  factors,  and  logistic  supportability) ,  that  all  significant  design 
problems  have  been  identified,  and  that  solutions  to  these  problems  are 
in  hand. 


The  Service  Operational  Test  and  Evaluation  Agency  conduct 
operational  assessments  to  estimate  the  system's  operational  effectiveness 
and  operational  suitability,  identify  needed  modifications,  and  provide 
information  on  tactics,  doctrine,  organization,  and  personnel  requirements. 
The  OT&E  program  is  accomplished  in  an  environment  as  operationally 
realistic  as  possible.  Typical  operational  and  support  personnel  are 
used  to  obtain  a  valid  estimate  of  the  user's  capability  to  operate  and 
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maintain  the  system.  The  user  of  the  system  monitors  test  and  evaluation 
during  the  Concept  Demonstration  and  Validation  Phase;  among  the  most 
important  products  of  user  monitoring  are  the  attainment  of  early 
orientation  and  advanced  training,  demonstrations  of  system  performance, 
and  valid  operational  test  (OT)  assessments  of  system  maintainability 
and  supportability. 

The  Development  Agency  prepares  the  results  of  demonstration 
and  validation  DT&E  in  report  form  for  review  by  the  Service  Headquarters 
and  the  Service  acquisition  review  council  prior  to  system  acquisition 
review  by  DoD.  The  report  includes  the  results  of  testing  with  supporting 
information,  conclusions,  and  recommendations  for  full-scale  development. 
At  the  same  time,  the  OT&E  Agency  prepares  independent  OT&E  assessments 
which  contain  estimates  of  the  system's  operational  effectiveness  and 
suitability.  OT&E  assessments  provide  a  permanent  record  of  OT&E 
accomplished,  an  audit  trail  of  OT&E  data,  test  results,  conclusions, 
and  recommendations.  This  information  is  used  to  support  the  development 
of  the  Decision  Coordinating  Paper  (DCP),  which  is  prepared  for  Milestone 
II  to  recommend  which  of  the  alternative  systems  studied  in  the 
demonstration  and  validation  phase  will  proceed  into  full-scale  engineering 
development. 

2.3.3  Test  Contributions  Prior  to  Milestone  III 

The  objective  of  the  Full-Scale  Development  (FSD)  Phase  prior 
to  Milestone  III  is  to  design,  fabricate  and  test  a  preproduction  system 
that  closely  approximates  the  final  product.  Test  and  evaluation 
activities  during  this  period  yield  much  useful  information.  For  example, 
data  obtained  during  FSD  test  and  evaluation  is  used  to  assist  in 
evaluating  the  system's  maintenance  training  requirements  and  In  evaluating 
the  proposed  training  program.  Test  results  generated  during  FSD  test 
and  evaluation  also  support  the  user  in  refining  and  updating  tactics. 

During  the  FSD  phase,  development  test  and  evaluation  is  conducted 
to  satisfy  the  following  objectives: 

(1)  To  assess  the  critical  technical  issues,  as  specified  in  program 

documents,  for  example: 

(a)  To  determine  how  well  the  development  contract 

specifications  have  been  met; 

(b)  To  identify  system  technical  deficiencies  and  appropriate 

corrective  actions; 

(c)  To  determine  whether  the  system  is  compatible  and 

Interoperable  with  existing  and  planned  equipment  or 
systems; 

(d)  To  estimate  the  reliability,  maintainability,  and 

availability  of  the  system  after  it  is  deployed; 
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(e)  To  determine  whether  the  system  Is  safe  and  ready  for 
operational  test  and  evaluation  (OT&E); 

(f)  To  validate  any  configuration  changes  caused  by  correcting 
deficiencies,  modifications,  or  product  Improvements; 

(g)  To  assess  human  factors  and  Identify  limiting  factors; 

(2)  To  assess  the  technical  risk  and  evaluate  the  tradeoffs  among 
specifications,  operational  requirements,  life  cycle  costs, 
and  schedules; 

(3)  To  assess  the  survivability,  vulnerability,  and  logistic 

supportabillty  of  the  system; 

(4)  To  verify  the  accuracy  and  completeness  of  the  technical 

documentation  developed  to  maintain  and  operate  the  weapons 
system; 

(5)  To  gather  information  for  training  programs  and  technical 

training  materials  needed  to  support  the  weapon  system; 

(6)  To  provide  information  on  environmental  Issues  to  be  used  in 

preparing  environmental  Impact  assessments;  and 

(7)  To  determine  system  performance  limitations  and  safe  operating 
parameters. 

Operational  test  and  evaluation  conducted  prior  to  the  production  decision 
at  Milestone  III  to  achieve  the  following: 

(1)  To  estimate  the  operational  effectiveness  and  suitability  of 
the  system; 

(2)  To  Identify  operational  deficiencies; 

(3)  To  recommend  and  evaluate  changes  In  production  configuration; 

(4)  To  provide  Information  for  developing  and  refining  logistics 
support  requirements  for  the  system  and  training,  tactics, 
techniques,  and  doctrine; 

(5)  To  provide  Information  to  refine  operation  and  support  (O&S) 
cost  estimates,  and  to  Identify  system  characteristics  or 
deficiencies  that  can  significantly  impact  O&S  costs; 

(6)  To  determine  whether  the  technical  publications  and  support 
equipment  are  adequate;  and 
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(7)  To  estimate  the  survivability  of  the  system  in  the  operational 
environment. 

Thus,  test  and  evaluation  activities  intensify  during  the  Full-Scale 
Development  Phase  and  make  significant  contributions  to  the  overall 
acquisition  decision  process. 

2.3.4  Test  Contributions  After  The  Production  Decision 

After  Milestone  III  when  the  production  decision  is  made,  T&E 
activities  continue  to  provide  important  insights.  Tests  described  in 
the  TEMP  and  not  completed  during  the  Full-Scale  Engineering  Development 
Phase  are  completed  during  the  Production  and  Deployment  Phase.  The 
residual  DT&E  is  usually  limited  to  all-weather  testing,  correction  of 
deficiencies,  and  engineering  modifications.  System  elements  are 
integrated  into  the  final  operational  configuration,  and  development 
testing  is  completed  when  the  system  performance  requirements  are  met. 
During  the  Production  Phase,  Government  representatives  normally  monitor 
or  conduct  Production  Acceptance  Test  and  Evaluation  (PAT&E).  Each  system 
is  verified  by  PAT&E  for  compliance  with  the  requirements  and 
specifications  of  the  contract. 

Post  production  testing  requirements  may  result  from  an 
acquisition  strategy  which  calls  for  block  changes  to  accommodate 
engineering  changes  or  the  use  of  pre-planned  product  improvements  (P3I) 
to  allow  parallel  development  of  high  risk  technology  and  modular  insertion 
of  system  upgrades  into  production  equipment.  Technology  breakthroughs 
and  significant  threat  changes  may  require  system  modifications.  The 
development  of  the  modifications  will  require  development  testing  and 
if  system  performance  is  significantly  changed,  then  operational  testing 
may  be  appropriate. 

Operational  test  and  evaluation  activities  continue  after  the 
production  decision  in  the  form  of  Follow-on  Operational  Test  and 
Evaluation.  The  initial  phase  of  FOT&E  may  be  conducted  by  either  the 
OT&E  Agency  or  the  user  commands,  depending  on  Service  directives.  It 
is  accomplished  to  verify  the  operational  effectiveness  and  suitability 
of  the  production  system  and  is  to  determine  if  deficiencies  identified 
during  Initial  Operational  Test  and  Evaluation  have  been  corrected. 
A  second  phase  of  FOT&E  is  conducted  by  the  user  to  refine  doctrine, 
tactics,  techniques,  and  training  programs  for  the  life  of  the  system. 

The  OT&E  Agency  prepares  a  final  report  at  the  conclusion  cf  its 
management  phase  of  FOT&E.  This  report  records  test  results,  describes 
the  evaluation  accomplished  to  satisfy  critical  issues  and  objectives 
established  for  FOT&E,  and  documents  its  assessment  of  deficiencies 
resolved  during  Full-Scale  Development.  Deficiencies  that  are  not 
corrected  are  recorded  with  recommended  corrective  actions. 
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A  final  report  on  FOT&E  is  also  prepared  by  the  using  command  test 
team  with  emphasis  on  the  operational  utility  of  the  system  when  operated, 
maintained,  and  supported  by  operational  personnel  using  the  concepts 
specified  for  the  system.  Specific  attention  is  devoted  to  the  following: 

(1)  The  degree  to  which  the  system  accomplishes  the  mission  when 
employed  by  operational  personnel  in  a  realistic  scenario  with 
the  appropriate  organization,  doctrine,  threat  (including 
countermeasures  and  nuclear  threats),  environment,  and  using 
tactics  and  techniques  developed  during  earlier  FOT&E; 

(2)  The  degree  to  which  the  system  can  be  placed  in  operational 

field  use,  with  specific  evaluations  of  availability, 

compatibility,  transportability,  interoperability,  reliability, 
wartime  usage  rates,  maintainability,  safety,  human  factors, 

manpower  supportability,  logistics  supportability,  and  training 
requirements; 

(3)  The  conditions  under  which  the  system  was  tested  including 
the  natural  weather  and  climatic  conditions,  terrain  effects, 
battlefield  disturbances  and  enemy  threat  conditions; 

(4)  The  ability  of  the  system  to  perform  its  required  functions 
for  the  duration  of  a  specified  mission  profile; 

(5)  System  weaknesses  such  as  the  vulnerability  of  the  system  to 

exploitation  by  countermeasures  techniques  and  the  practicality 
and  probability  of  an  adversary  exploiting  a  system 

susceptibility  in  combat. 

A  specific  evaluation  of  the  manpower  and  logistics  changes  needed  for 
the  effective  integration  of  the  system  into  the  user's  inventory  is 
also  made.  These  assessments  provide  essential  inputs  for  the  later 

phases  of  the  system  acquisition  cycle. 

2.4  SWMARY 

"Risk  management  is  the  means  by  which  the  program  areas  of 
vulnerability  and  concern  are  identified  and  managed."  (Reference  20). 
Test  and  evaluation  is  the  discipline  that  helps  to  illuminate  those 
areas  of  vulnerability.  The  importance  of  T&E  in  the  acquisition  process 
is  summarized  well  in  a  December  1986  report  produced  by  the  General 
Accounting  Office.  While  the  following  remarks  focus  on  operational 

test  and  evaluation,  they  also  serve  to  underscore  the  importance  of 
the  T&E  process  as  a  whole: 

OT&E  is  the  primary  means  of  assessing  weapon  system 
performance.  OT&E  results  are  important  in  making 
key  decisions  in  the  acquisition  process,  especially 
the  decision  to  proceed  from  full-scale  development 
to  production.  OT&E  results  provide  an  indication 
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of  how  well  new  systems  will  work  and  can  be  invaluable 
in  identifying  ineffective  or  unreliable  systems  before 
they  are  produced. 

Starting  production  before  adequate  OT&E  is  completed 
has  some  risks.  If  adequate  OT&E  is  not  done  and  the 
weapon  system  does  not  perform  satisfactorily  in  the 
field,  significant  changes  may  be  required.  Moreover, 
the  changes  will  not  be  limited  to  a  few  developmental 
models,  but  may  also  be  applied  to  items  already  produced 
and  deployed.  In  extreme  situations,  DoD  also  risks 
(1)  deploying  systems  which  cannot  adequately  perform 
significant  portions  of  their  missions,  thus  degrading 
our  deterrent/defensive  capabilities  and  (2)  endangering 
the  safety  of  military  personnel  who  operate  and  maintain 
the  systems. 
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CHAPTER  3 

TYPES  OF  TEST  AND  EVALUATION 


3.1  INTRODUCTION 

This  chapter  provides  an  overview  of  development  test  and 
evaluation  and  operational  test  and  evaluation  —  two  principal  types 
of  T&E;  it  also  discusses  the  role  of  qualification  testing  as  a  subelement 
of  development  testing.  Other  Important  types  of  T&E  are  also  Introduced. 
They  include:  multiservice  testing;  joint  test  and  evaluation;  live 
fire  testing;  nuclear,  chemical,  and  biological  testing;  and  nuclear 
hardening  and  survivability  testing.  As  Figure  3-1  illustrates, 
development  test  and  evaluation  and  operational  test  and  evaluation  are 
performed  throughout  the  acquisition  process  and  Identified  by  nomenclature 
that  reflects  the  phase  of  the  acquisition  cycle  in  which  they  occur. 

3.2  DEVELOPMENT  TEST  AND  EVALUATION 

Development  test  and  evaluation  (DT&E)  is  defined  In  DoD 
Directive  5000.3  as  “that  T&E  conducted  throughout  the  acquisition  process 
to  assist  in  engineering  design  and  development  and  to  verify  that 
technical  performance  specifications  have  been  met".  DT&E  is  planned 
and  monitored  by  the  developing  agency  and  Is  normally  conducted  by  the 
contractor.  However,  the  development  agency  may  perform  technical 
compliance  tests  prior  to  OT&E.  It  includes  the  T&E  of  components, 
subsystems,  preplanned  product  improvement  (P3I )  changes,  and 
hardware/software  integration,  as  well  as  preproduction  and  production 
qualification  testing.  It  encompasses  the  use  of  models,  simulations, 
and  test  beds,  as  well  as  prototypes  or  full-scale  engineering  development 
models  of  the  system.  DT&E  may  Involve  a  wide  degree  of  test  complexity 
,  depending  upon  the  type  of  system  or  test  article  under  development, 
e.g.,  tests  of  electronic  breadboards,  components,  subsystems,  or 
brassboards,  experimental  prototypes. 

DT&E  supports  the  system  design  process  through  a 
test-analyze-fix-retest  approach  that  involves  both  contractor  and 
government  personnel.  Because  contractor  testing  plays  a  pivotal  role 
In  the  total  test  program,  it  is  important  that  an  integrated  test  plan 
be  established  early  by  the  contractor  to  ensure  that  the  scope  of  the 
contractor's  test  program  satisfies  Government  test  objectives,  as  well 
as  contractor  objectives. 

The  Program  Manager  remains  responsible  for  the  ultimate  success 
of  the  overall  program.  He  and  the  test  specialists  on  his  staff  must 
foster  an  environment  that  provides  the  contractor  with  sufficient  latitude 
to  pursue  innovative  solutions  to  technical  problems,  and  at  the  same 
time,  provides  the  data  needed  to  make  rational  trade-off  decisions  between 
cost,  schedule,  and  performance  as  the  program  progresses. 


3-1 


PHASE  CONCEPT  CONCEPT  FULL  SCALE  LRIP  FULL-RATE  OPERATIONAL 

EXPLORATION  DEMONSTRATION  DEVELOPMENT  PRODUCTION/  I  SUPPORT  I 

DEFINITION  VALIDATION  DEPLOYMENT 


3-2 


Figure  3-1,  T&E  Phases  and  the  Acquisition  Process 


3.2.1  Preproduction  Qualification  Tests 

Qualification  testing  is  a  form  of  development  testing  that 

verifies  the  design  and  manufacturing  process.  Preproduction  qualification 
tests  are  formal  contractual  tests  which  confirm  the  integrity  of  the 
system  design  over  the  specified  operational  and  environmental  range. 

These  tests  usually  use  prototype  or  preproduction  hardware  fabricated 
to  the  proposed  production  design  specifications  and  drawings.  Such 
tests  include  contractual  reliability  and  maintainability  demonstration 
tests  required  prior  to  production  release.  Preproduction  Qualification 

Test  and  Evaluation  must  be  completed  before  Milestone  III. 

3.2.2  Production  Qualification  Tests 

Production  qualification  tests  are  conducted  on  production 

items  to  ensure  the  effectiveness  of  the  manufacturing  process,  equipment, 
and  procedures.  These  tests  are  conducted  on  each  item  or  a  sample  lot 
taken  at  random  from  the  first  production  lot,  and  are  repeated  if  the 
process  or  design  is  changed  significantly,  or  a  second  or  alternate 
source  is  brought  on  line.  These  tests  are  also  conducted  against 
contractual  design  and  performance  requirements. 

3.3  OPERATIONAL  TEST  AND  EVALUATION 

3.3.1  The  Difference  Between  Development  and  Operational  Testing 

Air  Force  Manual  55-43,  published  in  June  1979,  once  contained 
the  following  account  of  the  first  operational  test  and  evaluation;  this 
anecdote  serves  as  an  excellent  illustration  of  the  difference  between 
development  and  operational  testing: 

The  test  and  evaluation  of  aircraft  and  air  weapon 
systems  started  with  the  contract  awarded  to  the 
Wright  brothers  in  1908.  This  contract  specified 
a  craft  which  would  lift  two  men  with  a  total 
weight  of  350  pounds,  carry  enough  fuel  for  a 
flight  of  125  miles,  and  fly  40  miles  per  hour 
in  still  air.  The  contract  also  required  that 
testing  be  conducted  to  assure  this  capability. 

What  we  now  call  development  test  and  evaluation 
(DT&E)  was  satisfied  when  the  Wright  brothers 
(the  developer)  demonstrated  that  their  airplane 
could  meet  those  first  contract  specifications. 

However,  no  immediate  military  mission  had  been 
conceived  for  the  Wright  Flyer.  It  was  shipped 
to  Fort  Sam  Houston,  Texas,  where  Captain  Benjamin 
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D.  Foulols,  the  pilot,  had  orders  to  "teach  himself 
to  fly."  He  had  to  determine  the  airplane's 
performance,  how  to  ma Intaln  It,  and  the  kind 
of  organization  that  would  use  It.  Cavalry  wagon 
masters  had  to  be  trained  as  airplane  mechanics, 
and  Captain  Foulols  was  his  own  instructor  pilot. 

In  the  process.  Captain  Foulois  subjected  the 
Wright  Flyer  to  test  and  evaluation  under 
operational  conditions.  Foulois  soon  discovered 
operational  deficiencies.  For  example,  there 
was  no  seat  on  the  airplane.  During  hard  landings, 

Foulols'  130  pound  frame  usually  parted  company 
from  the  airplane.  To  correct  the  problem,  Foulois 
bolted  an  iron  tractor  seat  to  the  airplane. 

The  seat  helped,  but  Foulois  still  toppled  from 
his  perch  on  occasion.  As  a  further  improvement, 

Foulois  looped  his  Sam  Browne  belt  through  the 
seat  and  strapped  himself  in.  Ever  since  then, 
contoured  seats  and  safety  belts  —  a  product 
of  this  earliest  "operational"  test  and  evaluation 
-have  been  part  of  the  military  airplane. 

Captain  Foulois'  experience  may  seem  humorous 
now,  but  it  dramatically  illustrates  the  need 
for  operational  testing.  It  also  shows  that 
operational  testing  has  been  going  on  for  a  long 
time. 

As  shown  In  Table  3-1  where  development  testing  is  focused  on 
meeting  detailed  technical  specifications,  the  operational  test  focuses 
on  the  actual  functioning  of  the  equipment  in  realistic  combat  environment 
In  which  the  equipment  must  interact  with  men  and  peripheral  equipment. 
Where  DT&E  and  OT&E  are  separate  activities  and  are  conducted  by  different 
test  communities,  the  communities  must  interact  frequently  and  are  generally 
complementary.  DT&E  provides  a  view  of  the  potential  to  reach  technical 
objectives,  and  OTiE  provides  an  assessment  of  the  system's  potential 
to  satisfy  the  user  requirements. 

3.3.2  The  Purpose  of  Operational  Test  and  Evaluation 


Operational  Test  and  Evaluation  (OT&E)  is  conducted  "for  the 
purpose  of  determining  the  effectiveness  and  suitability  of  the  weapons, 
equipment,  or  munitions  for  use  in  combat  by  typical  military  users..." 
(DoD  Directive  5000.3). 

The  definitions  of  operational  effectiveness  and  operational 
suitability  are  outlined  in  DoDI  5000.2  are  as  listed  below: 
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Operational  Effectiveness.  The  overall  degree  of  mission 
accomplishment  of  a  system  when  used  by  representative  personnel  in  the 
environment  planned  or  expected  for  operational  employment  of  the  system 
considering  organization,  doctrine,  tactics,  survivability,  vulnerability, 
and  threat  (including  countermeasures,  nuclear,  and  chemical  and/or 
biological  threats). 

Operational  Suitability.  The  degree  to  which  a  system  can 
be  placed  satisfactorily  in  field  use  with  consideration  given  to 
availability,  compatibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors, 
manpower  supportability,  logistics  supportability,  documentation,  and 
training  requirements. 

In  each  of  the  Services,  operational  testing  is  conducted  under 
the  auspices  of  an  organization  that  is  independent  of  the  developer, 
in  as  operationally  realistic  environments  as  possible,  with  hostile 
forces  representative  of  the  anticipated  threat  and  with  typical  users 
operating  and  maintaining  the  system.  In  other  words,  "OT&E  is  conducted 
to  ensure  that  new  systems  meet  the  user's  requirements,  operate 
satisfactorily,  and  are  supportable  under  actual  field  conditions" 
(Reference  2).  The  major  questions  addressed  in  OT&E  are  shown  in  Figure 
3-2. 


Table  3-1.  Difference  Between  DT  &  OT 


DT 

Q1 

•  Controlled  by  Program  Manager 

Agency 

•  Controlled  by  Independent 

Agency 

•  One-on-One  Tests 

•  Many-on-Many  Tests 

•  Sterile  Controlled  Environment 

•  Tactical  Environment  with 
Operational  Scenario 

•  Contractor  Involvement 

•  No  Contractor  Involvement 

•  Trained  Experienced  Operators 

•  User  Troops  Recently  Trained 
on  Equipment 

•  Specific  Performance  Measurements 
and  Goals 

•  Operational  Ef fectiveness 
and  Suitability  Performance 
Measurements 
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Figure  3-2.  Major  Questions  Addressed  In  Operational  Test  and  Evaluation 


3.3.3  Pre-Production  Operational  Test  and  Evaluation 

OT&E  performed  before  the  full-rate  production  decision  is 
frequently  known  as  Initial  Operational  Test  and  Evaluation  (IOT&E). 
The  operational  assessment  normally  takes  place  during  the  concept 
exploration/definition  and  demonstration/validation  phases  and  is  used 
to  provide  an  early  assessment  of  potential  operational  effectiveness 
and  suitability  and  to  project  the  system's  potential  to  meet  the  user's 
requirements.  The  Initial  operational  test  begins  during  the  full-scale 
development  phase  and  ends  with  the  full-rate  production  decision.  This 
test  may  not  be  the  first  OT  conducted  on  the  system.  The  OT  is  conducted 
on  a  production  representative  system  using  typical  operational  personnel 
in  as  realistic  a  scenario  as  possible  to  verify  a  system's  operational 
effectiveness  and  suitability,  and  to  ensure  that  the  system  meets 
operational  thresholds. 

3.3.4  Follow-on  Operational  Test  and  Evaluation 

OT&E  performed  after  the  start  of  Full  Rate  Production  may 
be  known  as  Follow-on  Operational  Test  and  Evaluation  and  is  conducted 
during  production  and  deployment.  It  too  is  sometimes  divided  into  two 
separate  activities.  Preliminary  FOT&E  is  normally  conducted  after  the 
Initial  Operational  Capability  is  attained  in  order  to  assess  full  system 
capability.  It  is  conducted  by  the  OT&E  organization  to  verify  the 
correction  of  deficiencies,  if  required,  and  to  assess  system  training 
and  logistics  status.  Subsequent  FOT&E  is  conducted  on  production  items 
throughout  the  life  of  a  system.  The  results  are  used  to  refine  estimates 
of  operational  effectiveness  and  suitability;  to  update  training,  tactics, 
techniques,  and  doctrine;  to  identify  operational  deficiencies,  and  to 
Identify  the  need  for  modifications.  This  FOT&E  is  conducted  by  the 
operating  command. 

3.4  MULTISERVICE  TEST  AND  EVALUATION 

Multi  service  test  and  evaluation  is  that  T&E  conducted  on  a 
system  being  acquired  for  use  by  more  than  one  Service.  All  affected 
Services  and  their  respective  operational  test  agencies  participate  in 
the  planning,  conduct,  reporting,  and  evaluation  of  a  multiservice  test 
program.  One  Service  is  designated  the  lead  Service  and  is  responsible 
for  the  management  of  the  program.  The  lead  Service  is  charged  (by  DoD 
Directive  5000.3)  with  the  preparation  and  coordination  of  a  single  report 
that  reflects  the  system's  operational  effectiveness  and  suitability 
for  each  Service. 

The  management  challenge  in  a  multiservice  test  program  stems 
from  the  fact  that  the  items  undergoing  test  will  not  necessarily  be 
used  by  each  of  the  Services  for  identical  purposes.  Differences  between 
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the  Services  in  performance  criteria,  tactics,  doctrine,  configuration 
of  armament  or  electronics,  and  the  operating  environment  usually  exist. 
As  a  result,  a  deficiency  or  discrepancy  considered  disqualifying  by 
one  Service  is  not  necessarily  disqualifying  for  all  of  the  Services. 
It  is  incumbent  upon  the  lead  Service  to  establish  a  discrepancy  reporting 
system  that  permits  each  participating  Service  to  document  all 
discrepancies  noted.  At  the  conclusion  of  a  multiservice  T&E,  each 

participating  OT&E  agency  prepares  an  independent  evaluation  report  in 

its  own  format  and  submits  that  report  through  its  normal  Service  channels. 
The  lead  Service  OT&E  agency  prepares  the  documentation  that  goes  forward 
to  the  Defense  Acquisition  Board;  this  documentation  is  coordinated  with 
all  participating  OT&E  agencies. 

3.5  JOINT  TEST  AND  EVALUATION 

Joint  Test  and  Evaluation  is  not  the  same  as  multiservice  test 
and  evaluation.  Joint  test  and  evaluation  is  a  specific  program  activity 
sponsored  and  funded  by  the  Office  of  the  Secretary  of  Defense.  Joint 

T&E  programs  are  not  acquisition  oriented,  instead  are  a  means  of  examining 
joint  Service  tactics  and  doctrine.  Past  joint  test  programs  have  been 

conducted  to  provide  information  required  by  the  Congress,  by  the  Office 
of  the  Secretary  of  Defense,  by  the  commanders  of  the  Unified  and  Specified 
Commands,  and  by  the  Services.  Joint  tests  are  usually  characterized 
as  either  Joint  Deyelopment  T&E  or  Joint  Operational  T&E.  Joint 
development  tests  and  evaluations  focus  on  obtaining  information  in  the 
following  areas: 

o  System  Requirements 

o  System  Performance 

o  System  Interoperability 

o  Technical  Concepts 

o  Technical  Improvements 

o  Improved  Testing  Methodologies 
o  Test  Resource  Requirements 

Joint  operational  tests  and  evaluations  are  conducted  using  actual  fielded 
equipment,  simulators,  or  surrogate  equipment  in  an  exercise  or  operational 
environment  to  obtain  data  pertinent  to  operational  doctrine,  tactics, 
and  procedures. 

The  Office  of  the  Secretary  of  Defense  reviews  candidate 
nominations  for  joint  test  programs  each  year,  and  if  a  proposal  is  deemed 
appropriate  for  a  joint  test,  a  lead  Service  is  selected  and  tasked  to 
plan  and  execute  the  program  using  a  test  force  of  participating  Service 
personnel . 


The  commanders  of  the  four  Service  operational  test  agencies 
--the  Army  Operational  Test  and  Evaluation  Agency  (OTEA),  the  Navy 
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Operational  Test  and  Evaluation  Force  (OPTEVFOR),  the  Air  Force  Operational 
Test  and  Evaluation  Center  (AFOTEC),  and  the  Marine  Corps  Operational 
Test  and  Evaluation  Activity  (MCOTEA)  --  have  agreed  to  a  Memorandum 
of  Agreement  on  Multi -Service  OT&E  and  Joint  T&E  (Reference  37)  that 
stipulates  how  both  types  of  programs  are  to  be  managed. 

3.6  LIVE  FIRE  TESTING 

The  Live  Fire  Test  program  was  mandated  by  the  Congress  in 
the  National  Defense  Authorization  Act  for  Fiscal  1987  (Public  Law  99-661) 
passed  in  November  1986.  Specifically,  this  law  stipulates  that  a  major 
defense  acquisition  program  may  not  proceed  beyond  low-rate  initial 
production  until  realistic  survivability  or  (in  the  case  of  missiles 
and  munitions)  lethality  testing  has  been  completed.  Recently,  an 
amendment  has  been  proposed  to  substitute  the  word  "vulnerability"  for 
"survivability"  to  make  the  legislation  more  consistent  with  DOD  practice. 

In  1984,  prior  to  the  passage  of  this  legislation,  the  OSD 
had  chartered  a  joint  test  program  designed  to  address  similar  questions 
relative  to  systems  already  in  field  use.  This  program,  the  Joint  Live 
Fire  Test,  was  initially  divided  into  two  distinct  parts:  Armor/Antiarmor 
and  Aircraft.  The  program  has  the  following  objectives: 

o  Gather  empirical  data  on  the  vulnerability  of  existing 

U.S.  systems  to  Soviet  weapons; 

o  Gather  empirical  data  on  the  lethality  of  existing  U.S. 
weapons  against  Soviet  systems; 

o  Provide  Insights  Into  the  design  changes  necessary  to 

reduce  vulnerabilities  and  Improve  lethalities  of  existing 
U.S.  weapon  systems;  and 

o  Calibrate  current  vulnerability  and  lethality  models. 

The  recently-legislated  Live  Fire  Test  (LFT)  Program  complements 
the  older  Joint  Live  Fire  (JLF)  Program.  While  the  Joint  Live  Fire  (JLF) 
Program  was  designed  to  test  systems  already  fleYded  which  were  not 
completely  tested  when  they  were  developed,  the  spirit  and  Intent  of 
the  Live  Fire  Testing  (LFT)  Legislation  is  to  avoid  the  need  to  play 
"catch-up."  This  program  requires  the  Services  to  test  their  weapons 
systems  against  the  expected  combat  threat  as  early  as  possible  to  Identify 
design  characteristics  which  cause  undue  combat  damage,  or  measure 
munitions  lethality.  Remedies  for  deficiencies  can  entail  required 
retrofits,  production  stoppages  or  other  more  time-consuming  solutions. 
The  essential  feature  of  Live  Fire  Testing  is  that  appropriate  threat 
munitions  are  fired  against  a  major  U.S.  system  configured  for  combat 
to  test  its  vulnerability,  and/or  that  a  major  U.S.  munition  or  missile 
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is  fired  against  a  threat  target  configured  for  combat  to  test  the 
lethality  of  the  munition  or  missile. 

Live  Fire  Test  and  Evaluation  Guidelines  were  issued  by  the 
Director,  Live  Fire  Testing  in  May  1987  to  supplement  DoD  Test  and 
Evaluation  Master  Plan  guidelines  (DOD  5000.3-M-l)  in  areas  pertaining 
to  live  fire  testing  (Reference  34).  These  guidelines  encompass  all 

major  defense  acquisition  programs  and  define  LFT  requirements. 

3.7  NUCLEAR,  BIOLOGICAL,  AND  CHEMICAL  WEAPONS  TESTING 

The  testing  of  nuclear,  biological  and  chemical  (NBC)  weapons 

is  highly  specialized  and  regulated.  Program  managers  involved  in  these 
areas  are  advised  to  consult  authorities  within  their  chain  of  command 
for  the  specific  directives,  instructions,  and  regulations  that  apply 

to  their  individual  situations.  Nuclear  weapons  tests  are  divided  into 

categories  in  which  the  responsibilities  of  the  Department  of  Energy 
(DOE),  the  Defense  Nuclear  Agency  (DNA),  and  the  military  Services  are 
clearly  assigned,  with  DOE  responsible  for  nuclear  warhead  technical 
tests,  DNA  responsible  for  nuclear  weapons  effects  tests,  and  the  Services 
responsible  for  the  testing  of  Service-developed  components  of  nuclear 
subsystems.  All  nuclear  tests  are  conducted  within  the  provisions  of 
the  Limited  Test  Ban  Treaty  that  generally  restricts  nuclear  detonations 
to  the  underground  environment.  Nuclear  weapons  testing  requires  extensive 
coordination  between  Service  and  DOE  test  personnel  (Reference  55). 

Since  the  United  States  signed  and  ratified  the  Geneva  Protocol 
of  1925,  U.S.  policy  has  been  that  the  United  States  will  never  be  the 
first  to  use  lethal  chemical  weapons;  it  may,  however,  retaliate  with 
chemical  weapons  if  so  attacked.  With  the  signing  and  ratification  of 
the  1972  Biological  and  Toxin  Weapon  Convention,  the  United  States  formally 
adopted  the  position  that  it  would  not  employ  biological  or  to.  in  weapons 
under  any  circumstances.  All  such  weapons  were  destroyed  in  the  early 
1970s  (Reference  38). 

With  regard  to  the  retaliatory  capability  in  chemical  weapons, 
the  Service  secretaries  are  responsible  for  ensuring  that  their 
organizations  establish  requirements  and  determine  the  military 
characteristics  of  chemical  deterrent  items  and  chemical  defense  items. 
The  Army  has  been  designated  the  DoD  Executive  Agent  for  DoD  chemical 
warfare,  research,  development  and  acquisition  programs  (Reference  39). 

Unite  States  policy  on  chemical  warfare  seeks  to: 

o  Deter  the  use  of  chemical  warfare  weapons  by  other  nations; 

o  Provide  the  capability  to  retaliate  if  deterrence  fails; 

and 

o  Achieve  the  early  termination  of  chemical  warfare  at  the 

lowest  possible  intensity.  (Reference  39). 
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In  addition  to  the  customary  development  tests  (conducted  to 
determine  if  a  weapon  meets  technical  specifications)  and  operational 
tests  (conducted  to  determine  if  a  weapon  will  be  useful  in  combat), 
chemical  weapons  testing  involves  two  types  of  chemical  tests:  chemical 
mixing  and  biotoxicity.  Chemical  mixing  tests  are  conducted  to  obtain 
information  on  the  binary  chemical  reaction.  Biotoxicity  tests  are 
performed  to  assess  the  potency  of  the  agent  generated.  Chemical  weapons 
testing,  of  necessity,  relies  heavily  on  the  use  of  nontoxic  simulants 
since  such  substances  are  more  economical  and  less  hazardous,  and  since 
open  air  testing  of  live  agents  was  restricted  in  1969  (Reference  39). 

3.8  NUCLEAR  HARDNESS  AND  SURVIVABILITY  TESTING 

Nuclear  hardness  is  a  quantitative  description  of  the  physical 
attributes  of  a  system  or  component  that  will  allow  it  to  survive  in 
a  given  nuclear  environment.  Nuclear  survivability  is  the  capability 
of  a  system  to  survive  in  a  nuclear  environment  and  to  accomplish  its 
mission.  DoD  policy  requires  the  incorporation  of  nuclear  hardness  and 
survivability  features  in  the  design,  acquisition,  and  operation  of  major 
and  nonmajor  systems  that  must  perform  critical  missions  in  nuclear 
conflicts.  Nuclear  hardness  levels  must  be  quantified  and  validated. 
(Reference  12). 

The  test  and  evaluation  techniques  used  to  assess  nuclear 
hardness  and  survivability  include:  nuclear  testing,  physical  testing 
in  a  simulated  environment,  modeling,  simulation,  and  analysis.  Although 
nuclear  tests  provide  a  high  degree  of  fidelity  and  valid  results  for 
survivability  evaluation,  they  are  not  practical  for  most  systems  due 
to  cost,  long  lead  times,  and  international  treaty  constraints. 
Underground  testing  is  available  only  on  a  prioritized  basis  for  critical 
equipment  and  components  and  is  subject  to  a  frequently  changing  test 
schedule.  Physical  testing  provides  an  opportunity  to  observe  personnel 
and  equipment  in  a  simulated  nuclear  environment.  Modeling,  simulation 
and  analysis  are  particularly  useful  in  the  early  stages  of  development 
to  provide  early  projections  before  system  hardware  is  available.  These 
methods  are  also  used  to  furnish  assessments  in  area  that,  because  of 
safety  or  testing  limitations,  cannot  be  directly  observed  through  nuclear 
or  physical  testing. 

3.9  SUMMARY 

Test  and  evaluation  is  a  technique  used  to  address  critical 
questions  during  system  development.  These  questions  may  involve: 
technical  issues  (development  testing),  effectiveness,  suitability,  and 
supportability  issues  (operational  testing),  issues  affecting  more  than 
one  Service  (multiservice  and  joint  testing),  vulnerability  and  lethality 
issues  (live  fire  testing),  nuclear  survivability,  or  the  use  of  other 
than  conventional  (i.e.,  nuclear,  biological,  or  chemical)  weapons. 


CHAPTER  4 
EVALUATION 


4.1  INTRODUCTION 

This  chapter  describes  the  "evaluation"  portion  of  the  test 
and  evaluation  process.  It  stresses  the  importance  of  establishing  and 
maintaining  a  clear  audit  trail  from  system  requirements  through  critical 
issues,  evaluation  criteria,  test  objectives,  and  measures  of  effectiveness, 
to  the  evaluation  report.  The  importance  of  the  use  of  data  from  all 
sources  is  discussed  as  are  the  differences  in  approaches  to  evaluating 
technical  and  operational  data. 

4.2  DIFFERENCE  BETWEEN  "TEST"  AND  "EVALUATION" 

The  following  distinction  has  been  made  between  the  functions 
of  "test"  and  evaluation": 

While  the  terms  "test"  and  "evaluation"  are  most  often 
found  together,  they  actually  denote  clearly 
distinguishable  functions  in  the  RDT&E  process.  "Test" 
denotes  the  actual  testing  of  hardware/software  -models, 
prototypes,  production  equipment,  computer  programs 
—  to  obtain  data,  including  software,  valuable  in 
developing  new  capabilities,  managing  the  process, 
or  making  decisions  on  the  allocation  of  resources. 

"Evaluation"  denotes  the  process  whereby  data  are 
logically  assembled  and  analyzed  to  aid  in  making 
systematic  decisions.  (Reference  10) 


To  summarize,  evaluation  is  "the  review  and  analysis  of  qualitative  or 
quantitative  data  obtained  from  design  review,  hardware  inspection,  testing 
or  operational  usage  of  equipment"  (Reference  2). 


4.3  THE  EVALUATION  PROCESS 

The  evaluation  process  requires  careful  focus  on  the  development 
of  an  overall  test  and  evaluation  plan  that  will  provide  timely  answers 
to  critical  issues  and  questions  required  by  decision  authorities  as  part 
of  the  milestone  review  process. 

A  functional  block  diagram  of  a  generic  (i.e.,  not 
Service-specific)  evaluation  process  is  shown  in  Figure  4-1.  The  process 
begins  with  the  identification  of  a  deficiency  or  need  and  the  documentation 
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Figure  4-1.  Functional  Block  Diagram  of  the  Evaluation  Process 


of  an  operational  requirement.  It  continues  with  the  Identification  of 
critical  Issues  that  must  be  addressed  to  determine  If  the  system  meets 
the  requirement.  Criteria  must  then  be  established  to  define  required 
performance  or  supportability  thresholds  and  to  evaluate  progress  in 
reaching  them.  Test  and  evaluation  specialists  then  decompose  the  issues 
into  measurable  test  elements,  conduct  the  necessary  testing,  review  and 
analyze  the  test  data,  weigh  the  test  results  against  the  evaluation 
criteria,  and  prepare  an  evaluation  report  for  the  decision  authorities. 

4.4  ISSUES  AND  CRITERIA 

Issues  are  questions  regarding  a  system  that  require  answers 
during  the  acquisition  process.  Those  answers  may  be  needed  to  aid  in 
the  development  of  an  acquisition  strategy,  to  refine  requirements  and 
designs,  or  to  support  milestone  decision  reviews.  Criteria  are  the 
standards  by  which  the  issues  may  be  addressed  (Reference  62). 

4.4.1  Hierarchy  of  Issues 

As  Figure  4-2  illustrates,  issues  can  be  categorized  in  a 
hierarchical  system  according  to  the  subject  matter  they  address. 

4.4. 1.1  Systea  Program  Issues/Critical  Issues 

System  program  issues  are  often  known  as  "critical  issues." 
Critical  issues  are  defined  in  DOD  Manual  5000.3-M-l  as  "those  questions 
relating  to  a  system's  operational,  technical,  support  or  other  capability, 
that  must  be  answered  before  the  system's  overall  worth  can  be 
estimated/evaluated  and  that  are  of  primary  importance  to  the  decision 
authority  in  allowing  the  system  to  advance  to  the  next  acquisition  phase" 
(Reference  6).  System  program  issues  are  normally  identified  in  the  System 
Concept  Paper  (SCP),  Decision  Coordinating  Paper  (DCP),  and  requirements 
document.  The  system  development  and  production  baseline  documentation 
will  provide  much  of  the  performance  data  required  to  develop  the  issues. 

4.4. 1.2  Evaluation  Issues 

Evaluation  issues  are  those  issues  that  must  be  addressed  by 
technical  or  operational  evaluators  during  the  acquisition  process. 
Evaluation  Issues  are  separated  into  technical  and  operational  issues 
and  included  In  the  Test  and  Evaluation  Master  Plan  (TEMP). 

Technical  issues  primarily  concern  technical  characteristics 
or  engineering  specifications  and  are  normally  addressed  in  development 
testing.  Operational  Issues  concern  physical  characteristics  (weight, 
shape,  volume,  sturdiness)  and  operational  characteristics  (functions 
to  be  performed  by  equipment/concepts).  They  address  the  system's 
operational  effectiveness  and  suitability  when  examined  in  a  realistic 
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ACQUISITION  TEST  AND  EVALUATION,"  30  APRIL  1986. 


Figure  4-2.  Hierarchy  of  Issues 


operational  environment.  Evaluation  Issues  are  addressed  by  whatever 
means  necessary  (analysis/survey,  modeling,  simulation,  demonstration 
or  testing)  to  resolve  the  issue.  Issues  that  require  test  data  are  further 
referred  to  as  test  issues. 

4.4. 1.3  Test  Issues 

Test  issues  are  a  subset  of  evaluation  issues  and  address  areas 
of  uncertainty  that  require  test  data  to  resolve  the  issue  adequately. 
Test  issues  are  separated  into  technical  issues  that  are  addressed  by 
the  DT&E  community  and  operational  issues  that  are  addressed  by  the  OT&E 
community.  Test  issues  are  divided  Into  critical  and  non-crltlcal 
categories.  DoD  Directive  5000.3  requires  that  all  critical  test  and 
evaluation  issues,  objectives,  methodologies,  and  evaluation  criteria 
be  defined  during  the  initial  establishment  of  an  acquisition  program. 
Critical  test  issues  are  documented  in  the  TEMP.  The  directive  further 
requires  that  these  evaluation  criteria  serve  to  define  the  testing  required 
for  each  phase  of  the  acquisition  process  and  serve  as  the  structure  to 
guide  the  testing  program  (Reference  62). 

4.4.2  Criteria 

Criteria  are  statements  of  a  system's  required  technical 
performance  and  operational  effectiveness,  suitability,  and  supportability. 
Criteria  are  often  expressed  in  terms  of  "goals  and  thresholds".  (Some 
Services,  however,  specify  performance  and  supportability  requirements 
exclusively  in  terms  of  thresholds  and  avoid  the  use  of  the  concept  of 
goals.)  These  performance  measurements  provide  the  basis  for  collecting 
data  used  to  evaluate/answer  test  issues. 

Criteria  must  be  unambiguous  and  assessable  whether  stated 
qualitatively  or  quantitatively.  They  may  compare  the  mission  performance 
of  the  new  system  to  the  one  being  replaced,  compare  the  new  system  to 
a  predetermined  standard,  or  make  a  comparison  of  the  mission  performance 
results  from  using  the  new  system  to  not  having  the  system.  Criteria 
are  the  final  values  deemed  necessary  by  the  user.  As  such,  they  should 
be  developed  in  close  coordination  with  the  system  user,  other  testers, 
and  specialists  in  all  other  areas  of  operational  effectiveness  and 
suitability.  These  values  may  be  changed  as  systems  develop  and  associated 
testing  and  evaluation  proceeds.  For  instance,  a  Milestone  II  performance 
threshold  may  be  less  demanding  than  the  threshold  required  for  Milestone 
III  (Reference  61). 

4. 4. 2.1  Goals  and  Thresholds 

A  threshold  is  the  minimum  acceptable  level  of  performance 
required  by  a  test  article  or  system  to  perform  its  mission.  Thresholds 
are  stated  quantitatively  whenever  possible.  Specification  of  minimum 
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acceptable  performance  in  measurable  parameters  is  essential  to  the 
selection  of  appropriate  measures  of  effectiveness  which,  in  turn,  heavily 
influence  test  design.  Thresholds  are  of  value  only  when  actual  performance 
can  be  measured  against  them.  The  function  of  T&E  is  to  verify  the 
attainment  of  required  thresholds.  OPNAVINST  5000. 42C  states: 

T&E  is  the  major  control  mechanism  of  the  acquisition 
process.  Programs  advance  from  one  phase  to  the  next, 
not  by  the  calendar  or  planned  schedule,  but  by  actual 
achievement  of  present  thresholds,  verified  by  T&E 
(Reference  69), 


Goals  are  levels  of  performance  (established  by  the  user)  above 
that  required  which,  if  achieved,  will  provide  additional  operational 
capability.  Goals  are  not  normally  addressed  by  the  operational  tester 
whose  primary  concern  is  the  requirement.  While  goals  may  be  of  some 
value  to  the  developer  during  demonstration  and  validation,  their  relevance 
decreases  beyond  Milestone  II  when  most  system  performance  decisions 
have  been  made  and  their  utility  in  supporting  a  production  decision 
is  diminished.  However,  if,  on  occasion,  it  is  advantageous  to  the  user 
to  establish  goals  after  Milestone  II,  the  associated  evaluation  criteria 
must  be  clearly  identified  as  addressing  goals  and  not  requirements 
(Reference  83).  The  Navy  does  not  use  the  concept  of  goals  in  its 
acquisition  or  test  documentation. 

4. 4. 2.2  Evaluation  Criteria 

Evaluation  criteria  are  standards  by  which  the  achievement 
of  required  technical  and  operational  effectiveness/suitability 
characteristics  or  the  resolution  of  technical  or  operational  issues 
may  be  judged  (Reference  62).  Evaluation  criteria  are  associated  with 
objectives,  subobjectives,  and  measures  of  effectiveness  (MOEs).  For 
example,  an  MOE  (e.g.,  airspeed)  may  have  an  associated  evaluation 
criterion  (e.g.,  450  knots)  against  which  the  actual  performance  ( e.g ., 
425  knots)  is  compared  to  arrive  at  a  rating. 

Requirements,  thresholds,  and  goals  established  in  early  program 
documentation  form  the  basis  for  evaluation  criteria.  If  program 
documentation  is  incomplete,  the  tester  may  have  to  develop  evaluation 
criteria  in  the  absence  of  specific  requirements.  In  this  case,  the 
operating,  implementing,  and  supporting  commands  must  agree  to  the  criteria 
before  the  test  organization  makes  use  of  them  in  assessing  test  results. 
Ensuring  that  values  can  be  related  to  the  user's  operational  requirements 
is  a  most  important  consideration  when  identifying  and  establishing 
evaluation  criteria.  Testers  must  also  ensure  that  evaluation  criteria 
are  updated  if  requirements  change  (Reference  83). 
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4.5  EVALUATION  PLANNING 

4.5.1  Evaluation  Planning  Techniques 

Evaluation  planning  is  an  iterative  process  that  requires  formal 
and  informal  analyses  of  system  operation  (e.g.,  threat  environment, 
system  design,  tactics,  and  interoperability).  Techniques  that  have 
proved  effective  in  evaluation  planning  Include:  process  analysis 
techniques,  design  or  engineering  analysis  techniques,  matrix  analysis 
techniques,  and  dendritic  analysis  techniques  (Reference  61). 

4.5. 1.1  Process  Analysis  Techniques 

Process  analysis  techniques  consist  of  thinking  through  how 
the  system  will  be  used  in  a  variety  of  environments,  threats,  missions, 
and  scenarios  in  order  to  understand  the  events,  actions,  situations, 
and  results  that  are  expected  to  occur.  This  technique  aids  in  the 
identification  and  clarification  of  appropriate  measures  of  effectiveness 
(MOEs),  test  conditions,  and  data  requirements. 

4.5. 1.2  Design/Engineering  Analysis  Techniques 

Design  or  engineering  analysis  techniques  are  used  to  examine 
all  mechanical  or  functional  operations  that  the  system  has  been  designed 
to  perform.  These  techniques  involve  a  systematic  exploration  of  each 
hardware  and  software  component,  its  purpose,  its  performance  bounds, 
its  manpower  and  personnel  considerations,  known  problem  areas,  and  impact 
on  other  components.  Exploration  of  the  way  a  system  operates  compared 
to  the  functions  it  is  intended  to  perform,  often  identifies  issues, 
MOEs,  specific  data,  test  events,  and  required  instrumentation. 

4. 5. 1.3  Matrix  Analysis  Techniques 

Matrix  analysis  techniques  are  useful  for  analyzing  any  situation 
where  two  classification,  need  to  be  cross-referenced.  For  example, 
a  matrix  of  "Types  of  Data"  versus  "Means  of  Data  Collection"  can  reveal 
not  only  types  of  data  having  no  planned  means  of  collection,  but  also 
redundant  or  backup  collection  systems.  Matrix  techniques  are  useful 
as  checklists,  as  organizational  tools,  or  as  a  means  of  identifying 
and  characterizing  problem  areas.  Matrix  techniques  are  effective  for 
tracing  system  operational  requirements  through  contractual  specification 
documents  through  Issues  and  criteria  to  sources  of  individual  data  or 
specific  test  events. 

4. 5. 1.4  Dendritic  Analysis  Techniques 

Dendritic  analysis  techniques  are  an  effective  means  for 
decomposing  critical  Issues  to  the  pMnt  where  actual  data  requirements 
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and  test  measurements  can  be  identified.  In  these  techniques,  issues 
are  successively  broken  down  into  objectives,  subobjectives,  measures 
of  effectiveness,  and  data  requirements  in  a  root- like  structure  such 
as  that  depicted  in  Figure  4-3.  In  this  approach,  objectives  are  used 
to  clearly  express  the  broad  aspects  of  T&E  related  to  the  critical  issues 
and  the  overall  purpose  of  the  test.  Subobjectives  are  developed  as 
subsets  of  the  objectives  and  are  designed  to  treat  specific  and 
addressable  parts  of  the  objectives.  Each  subobjective  is  traceable 
as  a  direct  contributor  to  one  objective  and,  through  it,  is  identifiable 
as  a  direct  contributor  to  addressing  one  or  more  critical  issues 
(Reference  83).  Each  test  objective  and  subobjective  is  also  linked 
to  one  or  more  Measures  of  Effectiveness  (quantitative  or  qualitative 
measures  of  system  performance  under  specified  conditions)  which,  in 
turn,  are  tied  to  specific  data  elements.  The  dendritic  approach  has 
become  a  standard  military  planning  technique. 

4.5.2  Sources  of  Data 

As  evaluation  and  analysis  planning  matures,  focus  turns  toward 
identifying  data  sources  as  a  means  for  obtaining  each  data  element. 
Initial  identification  tends  to  be  generic  such  as:  engineering  study, 
simulation,  modeling,  or  contractor  test.  Later  identification  reflects 
specific  studies,  models,  and/or  tests.  A  data  source  matrix  is  a  useful 
planning  tool  to  show  where  data  are  expected  to  be  obtained  during  the 
T&E  of  the  system. 

There  are  many  sources  of  data  that  can  contribute  to  the 
evaluation.  Principal  sources  include:  studies  and  analyses;  models, 
simulations,  and  wargames,  contractor  testing,  development  testing, 
operational  testing,  and  comparable  systems. 

4.6  EVALUATING  DEVELOPMENT  AND  OPERATIONAL  TESTS 

Technical  and  operational  evaluations  employ  different  techniques 
and  have  different  evaluation  criteria.  DT&E  is  often  considered 
"technical"  evaluation  while  OT&E  addresses  the  operational  aspects  of 
a  system.  Technical  evaluation  deals  primarily  with  instrumented  tests 
and  statistically  valid  data.  An  operational  evaluation  deals  with 
operational  realism  and  the  uncertainties  of  combat  (Reference  76). 
DT&E  uses  technical  criteria  for  evaluating  system  performance.  These 
criteria  are  usually  parameters  that  can  be  measured  during  controlled 
DT&E  tests  that  are  particularly  important  to  the  developing  organization 
and  the  contractor  but  of  less  Interest  to  the  independent  operational 
tester.  The  operational  tester  focuses  on  Issues  such  as  demonstrating 
target  acquisition  at  useful  ranges  or  air  superiority  in  combat  or  the 
probability  of  accomplishing  a  given  mission.  For  example,  during  DT&E, 
firing  may  be  conducted  on  a  round-by-round  basis  with  each  shot  designed 
to  test  an  Individual  specification  or  parameter  with  other  parameters 
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Figure  4-3.  Dendritic  Approach  to  Test  and  Evaluation 


held  constant.  Such  testing  is  designed  to  measure  the  technical 
performance  of  the  system.  In  contrast,  in  OT&E  proper  technical 
performance  as  regards  individual  specification/parameters  is  de-emphasized 
and  the  environment  is  less  controlled.  The  OT&E  authority  must  assess 
whether,  given  this  technical  performance,  the  weapon  system  is 
operationally  effective  and  operationally  suitable  when  employed  under 
realistic  combat  (with  opposing  force)  and  environmental  conditions  by 
typical  personnel. 

The  emphasis  in  DT  is  strictly  on  the  use  of  quantitative  data 
to  verify  the  attainment  of  the  technical  specifications.  Quantitative 
data  are  usually  analyzed  using  some  form  of  statistics.  Versus  DT&E, 
qualitative  data  takes  on  increasing  importance  in  OT&E  when  effectiveness 
and  suitability  issues  are  being  explored.  Many  techniques  are  used 
to  analyze  qualitative  data.  They  range  from  converting  expressions 
of  preference  or  opinion  into  numerical  values  to  establishing  a  consensus 
by  committee.  For  example,  a  committee  may  assign  values  to  parameters 
such  as  "feel",  "ease  of  use",  "friendliness  to  the  user",  and  "will 
the  user  want  to  use  it"  on  a  scale  of  1  to  10.  Care  should  be  exercised 
in  the  interpretation  of  the  results  of  qualitative  evaluations  since, 
when  numbers  are  assigned  to  them,  the  meaning  of  such  things  as  the 
average  evaluation  and  its  standard  deviation  can  have  meanings  different 
from  quantitative  data  averages  and  standard  deviations. 

4.6.1  Technical  Evaluation 

The  Service's  materiel  development  organizations  are  usually 
responsible  for  oversight  of  all  aspects  of  DT&E,  including  the  technical 
evaluation.  The  objectives  of  a  technical  evaluation  are: 

o  To  assist  the  developers  by  providing  information  relative 
to  technical  performance;  qualification  of  components;  compatibility, 
interoperability,  vulnerability,  lethality,  transportability,  and 
survivability;  reliability,  availability,  and  maintainability  (RAM); 
manpower  and  personnel;  system  safety;  integrated  logistics  support; 
correction  of  deficiencies;  accuracy  of  environmental  documentation; 
and  refinement  of  requirements. 

o  To  ensure  the  effectiveness  of  the  manufacturing  process 
of  equipment  and  procedures  through  production  qualification  T&E. 

o  To  confirm  readiness  for  0T  by  ensuring  that  the  system 
is  stressed  beyond  the  levels  expected  in  the  0T  environment. 

o  To  provide  Information  to  the  decision  authority  at  each 
decision  point  regarding  a  system's  technical  performance  and  readiness 
to  proceed  to  the  next  phase  of  acquisition. 
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o  To  determine  the  system’s  operability  in  the  required 
climatic  and  realistic  battlefield  environments  to  include  natural, 
induced,  and  countermeasure  environments  (Reference  54). 

4.6.2  Operational  Evaluation 

The  independent  operational  test  and  evaluation  authority  is 
responsible  for  the  operational  evaluation.  The  objectives  of  an 
operational  evaluation  are: 

o  To  assist  the  developers  by  providing  information  relative 
to  operational  performance;  doctrine,  tactics,  training,  logistics;  safety; 
manpower,  technical  publications;  RAM;  correction  of  deficiencies;  accuracy 
of  environmental  documentation;  and  refinement  of  requirements. 

o  To  ensure  that  only  systems  that  are  operationally  effective 
and  suitable  are  delivered  to  the  operating  forces. 

o  To  provide  information  to  the  decision  authority  at  each 
decision  point  as  to  a  system's  operational  effectiveness  and  suitability, 
and  readiness  to  proceed  to  the  next  phase  of  acquisition. 

o  To  assess,  from  the  user's  viewpoint,  a  system's 
desirability,  considering  systems  already  fielded,  and  the  benefits  or 
burdens  associated  with  the  system  (Reference  84). 

4.7  SWfWRY 

A  primary  consideration  in  identifying  the  Information  to  be 
generated  by  a  test  and  evaluation  program  is  a  clear  understanding  of 
the  decisions  the  information  will  support.  The  importance  of  structuring 
the  T&E  program  to  support  the  resolution  of  critical  issues  cannot  be 
overemphasized.  It  is  the  responsibility  of  those  involved  in  the  test 
and  evaluation  process  to  ensure  that  the  proper  focus  is  maintained 
on  key  issues,  that  the  T&E  program  yields  information  on  critical 
technical  and  operational  issues,  that  all  data  sources  necessary  for 
a  thorough  evaluation  are  tapped,  and  that  evaluation  results  are 
communicated  in  an  effective  and  timely  manner.  The  evaluation  process 
is  evolutionary  throughout  the  acquisition  cycles. 
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CHAPTER  5 

TEST-RELATED  DOCUMENTATION 


5.1  INTRODUCTION 

During  the  course  of  a  defense  acquisition  program,  many  documents 
are  developed  that  have  significance  for  those  responsible  for  testing 
and  evaluating  the  system.  This  chapter  Is  designed  to  provide  an  overview 
of  these  documents. 

As  Figure  5-1  shows,  test-related  documentation  spans  a  broad 
range  of  materials.  It  includes  requirements  documentation  such  as  the 
Mission  Need  Statement  (MNS),  program  decision  documentation  such  as  the 
System  Concept  Paper  (SCP)  and  Decision  Coordinating  Paper  (DCP),  program 
management  documentation,  such  as  the  Acquisition  Strategy,  Baseline 
documentation,  the  System  Engineering  Management  Plan  (SEMP),  the  Integrated 
Logistics  Support  Plan  (ILSP),  and  the  Test  and  Evaluation  Master  Plan 
(TEMP).  Of  importance  to  the  Program  Manager  and  to  test  and  evaluation 
managers  are  additional  test  program  documents  such  as  specific  test 
designs,  test  plans,  outline  test  plans/test  program  outlines,  evaluation 
plans,  and  test  reports.  This  chapter  concludes  with  a  description  of 
the  End-of-Test  Phase  and  Low-Rate  Initial  Production  (LRIP)  Reports, 
two  special  purpose  T&E  status  reports  that  are  used  to  support  the 
milestone  decision  process. 

5.2  REQUIREMENTS  DOCUMENTATION 
5.2.1  Continuing  Mission  Area  Analyses 

DoDD  5000.1  requires  the  Services  to  conduct  continuing  mission 
analyses  of  their  assigned  areas  of  responsibility.  These  Mission  Area 
Analyses  (MAA)  may  result  in  recommendations  to  initiate  new  acquisition 
programs  to  reduce  or  eliminate  operational  deficiencies.  If  a  need  cannot 
be  met  through  changes  in  tactics,  strategy,  doctrine,  or  training  and 
a  materiel  solution  is  required,  the  needed  capability  is  described  in 
a  document  known  as  an  Operational  and  Organizational  (0&0)  Plan,  Required 
Operational  Capability  (ROC),  Army;  an  Operational  Requirement  (OR),  Navy; 
or  a  Statement  of  Operational  Need  (SON),  Air  Force.  When  the  cost  of 
a  proposed  acquisition  program  is  estimated  to  exceed  $200  million  for 
research,  development,  test  and  evaluation  or  $1  billion  for  procurement, 
it  is  considered  a  major  program  and  requires  a  "Mission  Need  Statement." 
The  MAA  is  completed  at  the  beginning  of  a  program  and  reviewed  at  the 
end  (Milestone  V)  to  evaluate  system  modifications  or  new  starts. 
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MISSION  NEED 
STATEMENT 


Figure  5-1.  Test-Related  Documentation 


5.2.2  Mission  Need  Statement 

The  Mission  Need  Statement  is  submitted  to  the  Defense  Acquisition 
Executive  either  prior  to  or  concurrent  with  the  annual  Program  Objectives 
Memorandum  (POM)  submission.  The  content  and  format  of  the  Mission  Need 
Statement  is  prescribed  by  DoD  Instruction  5000.2.  In  summary,  it  contains 
the  following: 

o  A  description  of  the  mission  need 

o  The  projected  threat 

o  Timing  and  priority  of  the  need 
o  Alternatives  for  meeting  the  need 

o  A  recommendation  concerning  the  feasibility  of  a  cooperative 
development  program  with  an  allied  nation 
o  An  assessment  of  the  technical  risk  involved 

o  Funding  implications 

o  Constraints 

o  A  proposed  acquisition  strategy 

The  Mission  Need  Statement  or  the  other  requirements  documents  are  of 
particular  value  to  the  tester  since  they  form  the  basis  for  the  initial 
identification  of  critical  issues  that  will  be  addressed  in  the  test 
program. 

5.3  PROGRAM  DECISION  DOCUMENTATION 

5.3.1  Acquisition  Decision  Memorandum 

Secretary  of  Defense  decisions  at  major  milestones  in  the 

acquisition  process  are  recorded  in  a  document  known  as  an  Acquisition 
Decision  Memorandum  (ADM).  The  ADM  documents  a  SECDEF  decision  on  a  Mission 
Need  Statement  at  Milestone  0,  on  a  System  Concept  Paper  (SCP)  at  Milestone 
I,  or  on  a  Decision  Coordinating  Paper  (DCP)  at  Milestones  II  and  III. 
In  conjunction  with  an  ADM,  the  SCP  and  DCP  are  also  primary  program 

guidance  documents  providing  goals/thresholds  for  systems  performance. 

5.3.2  System  Concept  Paper 

The  System  Concept  Paper  (SCP)  documents  the  results  of  the 

concept  exploration  phase  and  is  used  to  support  the  Milestone  I  decision. 
It  identifies  the  concepts  that  will  be  developed  further  in  the 

demonstration  and  validation  phase  and  provides  reasons  for  the  elimination 
of  previously  considered  alternative  concepts.  It  describes  the  proposed 
acquisition  strategy  and  establishes  broad  goals  and  thresholds  for  the 
system's  cost,  acquisition  schedule,  and  operational  effectiveness  and 

suitability.  The  purpose  and  content  of  the  SCP  are  set  forth  in  DoD 
Instruction  5000.2.  Test  managers  will  find  the  information  contained 
in  the  SCP  useful  in  scoping  the  overall  test  program  since  the  SCP 


5-3 


identifies  the  key  areas  of  technological  and  produclbility  risk  that 
must  be  reduced  by  research  and  development  and  validated  by  test  and 
evaluation  before  the  Milestone  II  decision  is  made. 

5.3.3  Decision  Coordinating  Paper 

The  Decision  Coordinating  Paper  (DCP)  is  used  as  a  milestone 
decision  supporting  document,  updated  for  subsequent  milestone  decisions. 

It  summarizes  the  results  of  the  demonstration  and  validation  phase  and 
is  used  to  support  the  Milestone  II  decision.  The  DCP  identifies  program 
alternatives  and  establishes  explicit  goals  and  thresholds  for  program 
cost,  schedule,  operational  effectiveness,  and  operational  suitability. 
The  DCP  is  updated  prior  to  Milestone  III  to  describe  program  changes 
since  Milestone  II  and  to  propose  revisions  in  goals  or  thresholds,  if 
required.  The  DCP  prepared  prior  to  Milestone  III  contains  a  discussion 
of  the  operational  test  and  evaluation  results  that  demonstrate  that  the 
system  is  ready  to  proceed  to  full -rate  production.  Instructions  for 
the  preparation  of  the  DCP  are  contained  in  DoD  Instruction  5000.2. 

5.4  PROGRAM  MANAGEMENT  DOCUMENTATION 

5.4.1  Acquisition  Strategy 

An  acquisition  strategy  must  be  formulated  at  the  outset  of 
a  development  program.  The  strategy  constitutes  a  broad  set  of  concepts 
that  provides  direction  and  control  for  the  overall  development  and 
production  effort.  The  Acquisition  Strategy  is  updated,  as  required, 
throughout  the  life  of  a  program.  The  level  of  detail  reflected  in  the 
Acquisition  Strategy  can  be  expected  to  increase  as  a  program  matures. 
The  Acquisition  Strategy  serves  as  a  conceptual  basis  for  formulating 
functional  plans  such  as  the  System  Engineering  Management  Plan,  Integrated 
Logistics  Support  Plan,  and  the  Test  and  Evaluation  Master  Plan. 

It  is  Important  that  T&E  interests  be  represented  as  the 
Acquisition  Strategy  is  formulated  because  the  Acquisition  Strategy  should: 

o  Provide  an  overview  of  the  T&E  planned  for  the  program, 
ensuring  that  adequate  T&E  is  conducted  prior  to  the 
production  decision; 

o  Discuss  plans  for  providing  adequate  quantities  of  test 

hardware; 

o  Describe  how  test  hardware  will  be  funded  "up  front;"  and 

o  Identify  test  and  evaluation  organizations  that  will  have 

T&E  responsibility  for  the  program. 
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5.4.2 


Baseline  Documentation 


The  Baseline  development  starts  with  Mission  Need  Statement 
at  Milestone  0.  It  is  included  in  Annex  B  of  the  System  Concept  Paper 
at  Milestone  I  and  transitions  to  a  Development  (MSI I )  and  Production 
( MS  III)  Baseline.  The  Baseline  Documentation  (DODI  5000.2)  is  used  to 
enhance  stability  and  control  cost  growth  of  selected  major  programs. 
When  the  baseline  is  signed  by  the  Program  Manager,  Service  Acquisition 
Executive,  and  Defense  Acquisition  Executive,  it  becomes  a  mechanism  to 
control  program  instabilities.  Baseline  documents  are  required  for  all 
programs  in  full-scale  development.  The  document  must  describe  the  systems 
requirements,  unit  and  development  cost,  and  milestone  schedule.  Programs 
in  production  are  required  to  have  a  baseline  document  consisting  of  system 
requirements,  total  program  cost,  and  production  schedules.  System  baseline 
documentation  is  a  good  source  of  information  on  systems  requirements 
and  program  milestones  for  the  tester. 

5.4.3  System  Engineering  Management  Plan 

The  Systems  Engineering  Management  Plan  (SEMP)  governs  the  system 
engineering  effort  and  serves  as  a  top-level  management  plan.  The  content 
of  the  SEMP  is  prescribed  in  MIL-STD-499A  and  described  in  detail  in  the 
Defense  System  Management  College  Systems  Engineering  Management  Guide. 
The  SEMP  consists  of  three  parts:  Technical  Program  Planning  and  Control, 
System  Engineering  Process,  and  Engineering  Specialty  Integration. 

The  SEMP  is  supported  by  a  number  of  specialty  plans  that  describe 
activities  in  specific  areas  (e.g..  Integrated  Logistics  Support  Plan 
and  Test  and  Evaluation  Master  Plan).  Program  Managers  and  test  managers 
need  to  make  extensive  use  of  the  SEMP  when  developing  and  updating  the 
TEMP. 


Care  should  be  exercised  to  avoid  inconsistencies  between  the 
SEMP  and  the  TEMP.  Technical  performance  measurement  parameters  stated 
in  the  SEMP  should  be  the  same  as  those  in  the  TEMP.  To  prevent 
inconsistencies,  and  ensure  that  the  tester's  needs  are  addressed,  the 
Program  Manager  should  coordinate  Requests  for  Proposals  with  test  managers 
before  the  RFPs  are  released. 

5.4.4  Integrated  Logistics  Support  Plan 

Integrated  Logistics  Support  ( ILS)  is  a  composite  of  all  support 
considerations  necessary  to  assure  the  effective  and  economical  support 
of  a  system  at  all  levels  of  maintenance  for  its  programmed  life  cycle 
(Reference  64).  The  Integrated  Logistics  Support  Plan  (ILSP)  describes 
the  overall  ILS  program  and  includes  ILS  requirements,  tasks,  and  milestones 
for  the  current  and  succeeding  phases  of  the  program.  The  ILSP  serves 
as  the  source  document  for  ILS  input  to  other  program  documentation  such 
as  the  Test  and  Evaluation  Master  Plan. 

Standards  and  procedures  for  logistic  support  analysis  are 
documented  in  MIL-STD-1388-1A.  This  standard  requires  that  test  and 
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evaluation  programs  be  planned  to  serve  the  following  three  logistics 
supportability  objectives: 

(1)  Provide  measured  data  for  input  into  system  level  estimates 
of  readiness,  operational  costs,  and  logistics  support 
resource  requirements; 

(2)  Expose  supportability  problems  so  that  they  can  be  corrected 
prior  to  deployment;  and 

(3)  Demonstrate  contractor  compliance  with  quantitative 
supportability  -  related  design  requirements. 

Development  of  an  effective  T&E  program  requires  close  coordination  of 
efforts  among  all  system  engineering  disciplines,  especially  those  involved 
in  logistics  support  analyses.  The  ILSP  should  be  developed  prior  to 
Milestone  I  to  provide  a  skeletal  framework  for  logistics  support  analysis 
and  to  Identify  Initial  logistics  testing  requirements  that  can  be  used 
as  input  to  the  Test  and  Evaluation  Master  Plan  to  support  ILS  development. 

5.5  TEST  PROGRAM  DOCUMENTATION 

5.5.1  Test  and  Evaluation  Master  Plan 

The  Test  and  Evaluation  Master  Plan  (TEMP)  is  the  basic  planning 
document  for  all  T&E  related  to  a  particular  major  system  acquisition. 
It  is  prepared  by  the  Program  Management  Office  with  the  operational  test 
information  provided  by  the  Service  Operational  Test  Organization.  It 
is  used  by  OSD  and  the  Services  for  planning,  reviewing,  and  approving 
T&E  programs  and  provides  the  basis  and  authority  for  all  other  detailed 
T&E  planning  documents.  The  TEMP  identifies  all  critical  technical 
characteristics  and  operational  issues  and  describes  the  objectives, 
responsibilities,  resources,  and  schedules  for  all  completed  and  planned 
T&E.  The  TEMP  is  required  by  DoD  Directive  5000.3;  guidelines  for  its 
preparation  are  found  in  DoD  5000.3-M-l. 

The  TEMP  is  a  living  document  that  must  address  all  changes 
to  critical  issues  associated  with  an  acquisition  program.  Major  changes 
in  program  requirements,  schedule,  or  funding  usually  result  in  a  change 
In  the  test  program.  Thus,  the  TEMP  must  be  reviewed,  and  updated  on 
an  annual  basis  and  prior  to  each  milestone  decision,  to  ensure  that  T&E 
requirements  are  current.  The  TEMP  is  the  primary  document  used  in  the 
OSD  review  and  decision  process  to  assess  the  adequacy  of  planned  testing 
and  evaluation.  As  such,  the  TEMP  must  be  of  sufficient  scope  and  content 
to  explain  the  entire  T&E  program.  The  key  topics  in  the  TEMP  are  shown 
in  Figure  5-2. 
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Figure  5-2.  Key  Elements  of  TEMP 


Each  TEMP  submitted  to  OSD  should  be  a  summary  document,  detailed 
only  to  the  extent  necessary  to  show  the  rationale  for  the  type,  amount, 
and  schedules  of  the  testing  planned.  It  must  relate  the  T&E  effort  clearly 
to  technical  risks,  operational  Issues  and  concepts,  system  performance, 
reliability,  availability,  maintainability,  logistic  objectives  and 
requirements,  and  major  decision  points.  It  should  summarize  the  testing 
accomplished  to  date  and  explain  the  relationship  of  the  various 
simulations,  subsystem  tests,  integrated  system  development  tests  and 
initial  operational  tests  which,  when  analyzed  in  combination,  provide 
confidence  in  the  system's  readiness  to  proceed  into  the  next  acquisition 
phase.  The  TEMP  must  address  the  T&E  to  be  accomplished  in  each  program 
phase,  with  the  next  phase  addressed  in  the  most  detail.  The  TEMP  is 
also  used  as  a  coordination  document  to  outline  each  test  and  support 
organization's  role  in  the  T&E  program  and  identify  major  test  facilities 
and  resources.  TEMPs  supporting  the  production  and  initial  deployment 
decision  must  include  the  T&E  planned  to  verify  the  correction  of 
deficiencies  and  to  complete  production  qualification  testing  and  follow-on 
OT&E. 


The  objective  of  the  OSD  TEMP  review  process  is  to  ensure  successful 
test  and  evaluation  programs  that  will  support  decisions  to  commit  resources 
at  major  milestones.  The  T&E  procedures  considered  during  the  TEMP  review 
process  are: 

(1)  Development  Test  and  Evaluation  (DT&E)  and  Operational 
Test  and  Evaluation  (OT&E)  are  initiated  early  to  assess 
and  reduce  risks  and  estimate  operational  potential. 

(2)  Critical  issues,  test  directives,  and  evaluation  criteria 
are  related  to  mission  need  and  established  well  before 
testing  begins. 

(3)  Provision  is  made  for  the  collection  of  sufficient  test 
data  with  appropriate  test  instrumentation  to  minimize 
subjective  judgment. 

(4)  OT&E  is  conducted  by  an  organization  independent  of  the 
developer  and  user. 

(5)  The  test  methodology  and  instrumentation  provide  a  mature 
and  flexible  network  of  resources  that  stress  (as  early 
as  possible)  the  weapon  system  in  a  variety  of  realistic 
environments. 

5.5.2  Evaluation  Plan 

The  Navy  and  Air  Force  include  evaluation  planning  within  the 
test  plan.  The  Army  develops  a  separate  plan  which  is  used  to  specify 
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the  evaluation  and  analysis  techniques  that  will  be  required  once  the 
test  data  has  been  collected  and  processed.  The  Evaluation  Plan  is  closely 
linked  to  the  Test  Design,  especially  the  statistical  models  on  which 
the  Test  Design  is  built. 

The  Army  requires  the  development  of  an  "Independent  Evaluation 
Plan"  by  both  a  technical  independent  evaluator  and  an  operational 
independent  evaluator. 

The  objective  of  the  Army's  Independent  Evaluation  Plan,  according 
to  AR  70-10,  is  to  "address  the  issues;  describe  the  evaluation  of  issues 
which  require  data  from  sources  other  than  test;  state  the  technical  or 
operational  issues  and  criteria;  identify  data  sources;  state  the  approach 
to  the  independent  evaluation;  specify  the  analytical  plan  and  identify 
program  constraints."  (Reference  54) 

Evaluation  plans  are  prepared  for  all  systems  in  development 
by  the  operational  evaluators,  during  concept  exploration,  in  coordination 
with  the  system  developer.  The  Army  Master  Evaluation  Plan  becomes  an 
annex  to  the  TEMP  and  is  updated  when  the  TEMP  is  revised.  It  identifies 
each  evaluation  issue  and  the  methodology  to  be  used  to  assess  it,  and 
specifies  requirements  for  exchange  of  information  between  the  development/ 
operational  testers  and  materiel  developers. 

5.5.3  Test  Design  Plan 

Of  critical  importance  to  test  designers  for  "major"  tests  is 
to  ensure  that  the  test  is  constructed  to  provide  useful  Information  in 
all  areas/aspects  which  will  lead  to  an  assessment  of  the  system 
performance.  For  example,  a  complicated,  even  ingenious,  test  which  does 
not  provide  the  Information  required  by  the  decision  makers,  is  in  many 
respects,  a  failed  endeavor.  Therefore,  the  process  of  developing  a  "Test 
Concept"  or  "Test  Design"  [the  distinction  between  these  vary  from 
organization  to  organization]  should  be  whether  the  test  will  provide 
the  information  required  by  the  decision  makers.  In  other  words,  "are 
we  testing  the  right  things  in  the  right  way?"... "and  are  our  evaluations 
meaningful?" 

The  Test  Design  Plan  is  statistical  and  analytical  in  nature 
and  should  perform  the  following  functions: 

(1)  Structure  and  organize  the  approach  to  testing  in  terms 
of  specific  test  objectives; 

(2)  Identify  key  measures  of  effectiveness  (MOEs); 

(3)  Identify  the  required  data  and  demonstrate  how  the  data 
will  be  gathered,  stored,  analyzed,  and  used  to  satisfy  the  MOEs; 

(4)  Indicate  whether  modeling  and  simulation  will  help  in  meeting 
test  objectives;  and 

(5)  Identify  the  number  and  type  of  test  events. 
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The  Test  Design  serves  as  a  foundation  for  the  more  detailed  Test  Plan 
and  specifies  the  test  objectives,  test  events.  Instrumentation,  test 
methodology,  data  requirements,  data  management  needs  and  analysis 
requirements. 

5.5.4  Test  Plan 

The  Test  Plan  Is  the  vehicle  that  translates  a  test  concept 
and  statistical/analytical  test  design  into  concrete  resources,  procedures, 
and  responsibilities.  The  size  and  complexity  of  a  test  program  and  Its 
associated  test  plan  are  determined  by  the  nature  of  the  system  being 
tested  and  the  type  of  testing  that  is  to  be  accomplished.  Some  major 
weapons  systems  may  require  large  numbers  of  separate  tests  to  satisfy 

test  objectives,  and  thus  require  a  multi-volume  Test  Plan,  while  other 
testing  may  be  well  defined  by  a  relatively  brief  Test  Plan.  The  test 

plan  also  provides  a  description  of  the  equipment  configuration  and  known 
limitations  to  the  scope  of  testing.  The  type  of  information  typically 
Included  In  a  Test  Plan  Is  shown  In  Table  5-1. 

5.5.5  Outline  Test  Plan/Test  Program  Outline 

The  Army's  Outline  Test  Plan  (OTP)  and  Air  Force's  Test  Program 
Outline  (TPO)  are  essential  test  planning  documents.  These  documents 
are  formal  resource  documents  that  specify  the  resources  that  will  be 
required  to  support  the  test.  It  is  Important  that  these  documents  be 

kept  current  to  reflect  maturing  resource  requirements  as  the  test  program 
develops,  since  the  OTP  or  TPO  provides  the  means  for  programming  the 

necessary  resources.  The  Navy  makes  extensive  use  of  the  TEMP  to  document 
TiE  resource  requirements. 

5.5.6  Test  Reports 

5. 5.6.1  Quick-Look  Reports 

Quick-look  analyses  are  expeditious  analyses  performed  during 

testing  using  parts  of  data  set.  Such  analyses  are  often  used  to  assist 
in  the  management  of  test  operations.  Quick-look  reports  are  occasionally 
used  to  Inform  higher  authorities  of  test  results.  Quick-look  reports 

may  have  associated  briefings  that  present  T4E  results  and  substantiate 
conclusions  or  recommendations.  Quick  Look  reports  may  be  generated  by 
the  contractor  or  government  agency.  They  are  of  particularly  critical 
Interest  for  high  visibility  systems  which  may  be  experiencing  some 
development  difficulties. 

5.5.6.2  Final  Test  Report 

The  Final  Test  Report  disseminates  the  test  information  to 
decision  authorities,  program  office  staff,  and  the  acquisition  community. 
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TABLE  5-1  Sample  Test  Plan  Contents 


PRELIMINARY  PAGES 

i.  TITLE  PAGE 

ii.  ABSTRACT 

iii.  TABLE  OF  CONTENTS 

iv.  TERMS  AND  ABBREVIATIONS 

v.  *  RELATED  DOCUMENTS 

*THE  ACTUAL  NUMBER  OF  THESE  PAGES  WILL  BE  DETERMINED  BY  THE  LENGTH  OF 
PRELIMINARY  ELEMENTS  (e.g.,  TABLE  OF  CONTENTS,  TERMS  AND  ABBREVIATIONS,  ETC.). 

MAIN  BODY 

1.  INTRODUCTION 

2.  TEST  PURPOSE  AND  OBJECTIVES 

3.  CONCEPT  OF  TEST  OPERATIONS 

4.  METHOD  OF  ACCOMPLISHMENT 

5.  TEST  SCHEDULE 

6.  TEST  MANAGEMENT  AND  ORGANIZATION 

7.  RESPONSIBILITIES/SUPPORT 

8.  PERSONNEL 

9.  REQUIRED  TEST  REPORTS 

10.  SAFETY 

11.  SECURITY 

12.  INFORMATION 

13.  ENVIRONMENTAL  PROTECTION 

ANNEXES 

A.  TEST  DESIGN 

B.  DATA  REQUIREMENTS 

C.  INSTRUMENTATION  PLAN 

D.  LOGISTICS  SUPPORT  REQUIREMENTS 

E.  RELIABILITY  AND  MAINTAINABILITY  DATA  PLAN 

F.  I NTELLI G  E  NCE/TH  RE  AT  INFORMATION 

G.-Z.  AS  REQUIRED 

1.  2,  3.  ETC.  DETAILED  TEST  PROCEDURES  (NAME  OF  TEST) 

DISTRIBUTION: 


Source:  "Standard  Procedures  for  USAF  OT&E,"  July,  1974. 
*7-1 -MCL2 -000584-1 0 
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It  provides  a  permanent  record  of  the  execution  of  the  test  and  its  results. 
The  Final  Test  Report  should  relate  the  test  results  to  the  critical  issues 
and  address  the  objectives  stated  in  the  Test  Design  and  Test  Plan.  A 
Final  Test  Report  may  be  separated  into  two  sections  -  a  main  section 
providing  the  essential  information  about  test  methods  and  results,  and 
a  second  section  consisting  of  supporting  appendices  to  provide  details 
and  supplemental  information.  Generally,  the  following  topics  are  included 
in  the  main  body  of  the  report: 

(1)  Test  Purpose 

(2)  Issues  and  Objectives 

(3)  Method  of  Accomplishment 

(4)  Results  (keyed  to  the  objectives  and  issues) 

(5)  Discussion,  conclusions,  and  recommendations. 

Appendices  of  the  Final  Test  Report  may  address  the  following  topics: 

(1)  Detailed  test  description 

(2)  Test  environment 

(3)  Test  organization  and  operation 

(4)  Instrumentation 

(5)  Data  collection  and  management 

(6)  Test  data 

(7)  Data  analysis 

(8)  Modeling  and  simulation 

(9)  Reliability,  availability,  and  maintainability  information 

(10)  Personnel 

(11)  Training 

(12)  Safety 

(13)  Security 

(14)  Funding 

(15)  Asset  Disposition 

The  Final  Test  Report  may  contain  an  evaluation  and  analysis 

of  the  results,  or  the  evaluation  may  be  issued  separately.  The  analysis 
tells  what  the  results  are,  whereas  an  evaluation  tells  what  the  results 
mean.  The  evaluation  builds  on  the  analysis  and  generalizes  from  it, 
showing  how  the  results  apply  outside  the  test  arena.  It  shows  what  the 
implications  of  the  test  are  and  may  provide  recommendations.  The 
evaluation  may  make  use  of  independent  analyses  of  all  or  part  of  the 

data;  it  may  employ  data  from  other  sources  and  it  may  use  modeling  and 

simulation  to  generalize  the  results  and  extrapolate  to  other  conditions. 
In  the  case  of  the  Army,  a  separate  Independent  Evaluation  Report  is  also 
prepared  by  both  technical  independent  evaluators,  and  operational 

Independent  evaluators. 
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5.6  OTHER  TEST-RELATED  STATUS  REPORTS 

5.6.1  End  of  Test  Phase  Report 

The  Servfces  are  required  by  DoD  Directive  5000.3  to  submit 
to  OSD  copies  of  their  formal  DT&E  and/or  OT&E  reports  that  are  prepared 
at  the  end  of  each  phase  of  DT&E  or  OT&E.  In  the  case  of  extended  test 
phases.  Interim  reports  must  be  submitted  at  least  annually.  For  major 
defense  acquisition  programs,  such  reports  must  be  received  by  the  Director 
of  Operational  Test  and  Evaluation  or  the  Deputy  Director  Defense  Research 
and  Engineering  (Test  and  Evaluation)  no  later  than  45  days  prior  to  a 
milestone  decision. 

5.6.2  Low-Rate  Initial  Production  Report 

Before  proceeding  beyond  Low-Rate  Initial  Production  (LRIP) 
for  each  major  defense  acquisition  program,  the  Director  of  Operational 
Test  and  Evaluation  must  report  to  the  Secretary  of  Defense  and  the  Senate 
and  House  of  Representatives  Committees  on  Armed  Services  and  on 
Appropriations.  This  report  addresses  whether  the  OT&E  performed  was 
adequate,  and  whether  the  OT&E  results  confirm  that  the  items  or  components 
actually  tested  are  effective  and  suitable. 

5.7  SUMMARY 

A  wide  range  of  documentation  is  available  to  the  test  manager 
and  should  be  used  to  develop  test  and  evaluation  programs  that  address 
all  relevant  issues.  The  Program  Manager  must  work  to  ensure  that  T&E 
requirements  are  considered  at  the  outset,  when  the  Acquisition  Strategy 
is  formulated.  He  must  also  require  early  and  close  coordination  and 
a  continuing  dialogue  among  those  responsible  for  the  Systems  Engineering 
Management  Plan,  the  Integrated  Logistics  Support  Plan,  and  the  Test  and 
Evaluation  Master  Plan. 
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CHAPTER  6 

NONDEVELOPMENT  ITEMS 


6.1  INTRODUCTION 

Many  options  are  available  when  an  acquisition  strategy  for 
a  new  system  is  chosen.  They  range  from  a  traditional  new  research  and 
development  program  to  the  use  of  "off-the-shelf"  nondevelopment  items 
(NDI).  Between  these  two  extremes  lie  other  acquisition  strategies  that 
call  for  using  nondevelopment  items  to  various  extents.  Figure  6-1, 
an  adaptation  of  an  illustration  found  In  Army  Materiel  Command  Pamphlet 
70-2,  shows  the  broad  spectrum  of  approaches  that  can  be  taken  in  a  system 
acquisition  and  provides  examples  of  systems  that  have  been  developed 
using  each  approach. 

6.1.1  Definition  of  NDI 

NDI  refers  to  materiel  available  from  a  variety  of  sources, 
but  involving  little  or  no  development  effort.  It  includes  commercial 
products,  materiel  developed  by  other  U.S.  Government  sources,  or  materiel 
developed  in  other  countries.  All  such  systems  are  required  to  undergo 
technical  and  operational  T&E  prior  to  the  procurement  decision,  unless 
a  definitive  decision  is  made  by  the  decision  authority,  that  previous 
testing  or  other  data  (such  as  user/market  investigations)  provide 
sufficient  evidence  of  acceptability  (Reference  54.) 

6.1.2  Advantages  and  Disadvantages  of  the  NDI  Approach 

The  use  of  NDI  offers  the  following  advantages: 

o  The  time  to  field  a  system  is  greatly  reduced,  and  a  quick 
response  is  provided  to  the  user's  needs; 

o  Research  and  development  costs  are  reduced;  and 

o  State-of-the-art  technology  is  available  immediately. 

NDI  offers  the  following  disadvantages: 

o  NDI  acquisitions  are  difficult  to  standardize  with  the  current 
fleet  equipment; 

o  NDI  acquisitions  create  logistics  support  difficulties; 

o  NDI  acquisitions  tend  to  not  have  competition  and  therefore, 
the  availability  of  second  source  is  not  present;  and 
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Figure  6-1.  The  Spectrum  of  Acquisition  Strategies 


o  With  NDI  acquisitions  engineering  and  test  data  is  often 
not  availabe. 

6.1.3  Types  of  NDI 

Nondevelopment  items  can  be  separated  into  two  general 
categories;  each  requires  a  modified  testing  approach.  The  categories 
are: 


(1)  Off-the-shelf  items  for  use  in  the  same  environment  for 
which  the  items  were  designed.  Such  items  normally  do 
not  require  development  testing  prior  to  the  production 
qualification  test  except  in  those  cases  where  a  contract 
may  be  awarded  to  a  contractor  who  has  not  previously 
produced  an  acceptable  finished  product  and  the  item  is 
assessed  as  high  risk.  In  that  case,  preproduction 
qualification  testing  would  be  required  (Reference  54.) 

(2)  Off-the-shelf  items  for  use  in  an  environment  other  than 
that  for  which  the  items  were  designed.  Such  items  may 
require  modifications  in  hardware  and/or  software.  These 
items  require  testing  in  an  operational  environment, 
preproduction  qualification  testing  (if  previous  testing 
resulted  in  item  redesign),  and  production  qualification 
testing. 

Existing  components  that  must  be  integrated  with  a  new  system  configuration 
may  be  purchased  off  the  shelf.  These  would  not  be  classified  as  a  NDI 
but  many  of  the  testing  and  evaluation  methods  would  still  apply.  This 
type  of  NDI  effort  requires  more  extensive  research,  development,  and 
testing  to  achieve  the  desired  system  configuration.  Testing  required 
includes:  feasibility  testing  in  a  military  environment;  preproduction 
qualification  testing;  hardware/software  integration  testing;  operational 
testing;  and  production  qualification  testing. 

Given  the  variety  of  NDI  approaches  that  may  be  employed,  it 
is  Imperative  that  the  acquisition  strategy  clearly  specify,  with  the 
agreement  of  the  testing  authority,  the  level  of  testing  that  will  be 
performed  on  NDI  systems  and  the  environment  those  systems  will  be  tested 
in. 


6.2  MARKET  INVESTIGATION  AND  PROCUREMENT 

A  market  investigation  is  the  central  activity  leading  to  the 
milestone  I  review  decision  regarding  the  use  of  an  NDI  acquisition 
strategy.  The  purpose  of  the  market  investigation  is  to  determine  the 
nature  of  available  products  and  the  number  of  potential  vendors.  Market 
investigations  may  vary  from  informal  telephone  inquiries  to  comprehensive 
industry  -  wide  reviews.  During  the  market  investigation,  sufficient 
data  must  be  gathered  to  support  a  definitive  NDI  decision,  to  finalize 
the  requirements,  and  to  develop  an  acquisition  strategy  that  is  responsive 
to  these  requirements. 
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During  the  market  investigation  phase,  a  formal  "request  for 
information"  process  may  be  followed  wherein  a  brief  narrative  description 
of  the  requirement  is  published  and  interested  vendors  are  invited  to 
respond,  test  samples  or  test  items  may  be  leased  or  purchased  at  this 
time  to  support  the  conduct  of  operational  suitability  tests,  to  evaluate 
the  ability  of  the  equipment  to  satisfy  the  requirements,  and  to  help 
build  the  functional  purchase  description  or  system  specification.  This 
type  of  preliminary  testing  should  not  be  used  to  select  or  eliminate 
any  particular  vendor  or  product  unless  it  is  preceded  by  competitive 
contracting  procedures  (Reference  61.) 

It  is  imperative  that  technical  and  operational  evaluators 
become  involved  during  this  early  stage  of  an  NDI  procurement,  that  they 
perform  an  early  assessment  of  the  initial  issues,  relate  these  issues 
to  test  and  evaluation  criteria,  and  provide  their  independent  evaluation 
plans  and  reports  to  the  decision  authorities  prior  to  the  milestone 
I  decision  review. 

6.3  NDI  TESTING 

6.3.1  General  Considerations 

Test  and  evaluation  must  be  considered  throughout  the  acquisition 
of  a  system  that  involves  NDI.  The  Program  Manager  and  his  staff  must 
ensure  that  the  testing  community  is  fully  involved  in  the  acquisition 
from  the  start.  The  amount  and  level  of  testing  required  depends  on 
the  nature  of  the  NDI  and  its  anticipated  use  and  should  be  planned  to 
support  the  design  and  decision  process.  Available  test  results  from 
all  commercial  and  Government  sources  will  determine  the  actual  extent 
of  testing  necessary.  There  are  some  inherent  advantages  in  NDI  testing. 
For  example,  a  NDI  usually  encompasses  a  mature  design.  The  availability 
of  this  mature  design  contributes  to  the  rapid  development  of  the  logistics 
support  system  that  will  be  needed.  In  addition,  there  are  more 
"production"  items  available  for  use  in  a  test  program.  The  Program 
Manager  and  his  staff  must  bear  in  mind  that  NDI  systems  also  require 
activity  in  areas  associated  with  traditional  development  and  acquisition 
programs.  For  example,  training  and  maintenance  programs  and  manuals 
must  be  developed  and  sufficient  time  should  be  allowed  for  their 
preparation. 

When  the  solicitation  package  for  an  NDI  acquisition  is  assembled, 
the  Program  Manager  must  ensure  that  it  includes  the  following  TAE-related 
i terns : 


(1)  Approved  test  and  evaluation  issues  and  criteria; 
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(2)  A  requirement  that  the  offeror  provide  a  description  of 
the  testing  performed  by  the  contractor  on  the  system, 
Including  test  procedures  followed,  test  data,  and  results 
achieved; 

(3)  Production  Qualification  Test  and  Quality  Conformance 
Requirements;  and 

(4)  Acceptance  test  plans  for  the  system  and  its  components. 

6.3.2  Testing  Before  Milestone  I 

An  Important  advantage  of  using  an  NDI  acquisition  strategy 
is  reduced  acquisition  time.  Consequently,  it  is  important  that  testing 
not  be  redundant  and  that  it  is  limited  to  the  minimum  effort  necessary 
to  obtain  the  required  data.  Testing  can  be  minimized  by: 

(1)  Obtaining  and  assessing  contractor  test  results; 

(2)  Obtaining  usage/failure  data  from  other  customers; 

(3)  Observing  contractor  testing; 

(4)  Obtaining  test  results  from  independent  test  organizations 
(e.g.,  Underwriter's  Laboratory);  and 

(5)  Verifying  selected  contractor  test  data. 

If  after  the  initial  data  collection  from  the  above  sources,  it  is 
determined  that  more  information  is  needed,  NDI  candidates  may  be  bought 
or  leased  and  technical  and  operational  tests  may  be  conducted. 

6.3.3  Testing  After  Milestone  I 

All  testing  to  be  conducted  after  the  initial  milestone  decision 
to  proceed  with  the  NDI  acquisition  should  be  described  in  the  Decision 
Coordinating  Paper  and  the  Test  and  Evaluation  Master  Plan.  Technical 
testing  is  conducted  only  if  specific  information  is  needed  that  cannot 
be  satisfied  by  contractor  or  other  test  data  sources.  Operational  testing 
Is  conducted  as  needed.  The  independent  operational  test  and  evaluation 
agency  should  concur  in  any  decisions  to  limit  or  eliminate  operational 
testing. 


Test  and  evaluation  continues  even  after  the  system  has  been 
fielded.  This  testing  takes  the  form  of  a  follow-on  evaluation  to  validate 
and  refine:  operating  and  support  cost  data;  reliability,  availability, 
and  maintainability  characteristics;  logistic  support  plans;  and  training 
requirements,  doctrine  and  tactics. 
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6.4 


RESOURCES  AND  FUNDING 


Programming  and  budgeting  for  an  NDI  acquisition  present  a 
special  challenge.  Because  of  the  short  duration  of  the  NDI  acquisition 
process,  the  standard  lead  times  required  in  the  normal  Planning, 
Programming,  and  Budgetary  System  (PPBS)  cycle  may  be  unduly  restrictive. 
This  situation  can  be  minimized  through  careful  advanced  planning  and, 
in  the  case  of  urgent  requirements ,  reprogramming/supplemental  funding 
techniques. 

Research,  Development,  Test  and  Evaluation  (RDT&E)  funds  are 
normally  used  to  support  the  conduct  of  the  market  investigation  phase 
and  the  purchase  or  lease  of  NDI  candidates  required  for  test  and 
evaluation  purposes.  RDT&E  funds  are  also  used  to  support  test  and 
evaluation  activities  such  as:  modification  of  the  NDI;  the  purchase 
of  specifications,  manufacturer's  publications,  repair  parts,  special 
tools  and  equipment;  the  transportation  of  the  NDI  to  and  from  the  test 
site;  and  the  training,  salaries  and  temporary  duty  (TDY)  costs  of  test 
and  evaluation  personnel.  Procurement  and  Operations  and  Maintenance 
funds  are  usually  used  to  support  production  and  deployment  costs. 

One  of  the  chief  advantages  of  using  an  NDI  acquisition  strategy 
is  reduced  overall  cost.  Additional  cost  savings  can  be  achieved  after 
a  contract  has  been  awarded  if  the  Program  Manager  ensures  that  incentives 
are  provided  to  contractors  to  submit  value  engineering  change  propose  Is 
to  the  Government  when  unnecessary  costs  are  identified. 

6.5  SUMMARY 

The  use  of  nondevelopment  items  in  a  system  acquisition  can 
provide  considerable  savings  in  both  time  and  cost.  The  testing  approach 
used  for  an  NDI  acquisition  must  be  carefully  tailored  to  the  type 
of  system  and  the  amount  of  test  data  already  available.  The  test  and 
evaluation  community  must  get  involved  early  in  the  process  so  that  all 
test  issues  are  adequately  addressed  and  timely  comprehensive  evaluations 
are  provided  to  decision  authorities. 


6-6 


MODULE  II 


DEVELOPMENT  TESTING 


Materiel  acquisition  is  an  iterative  process  of  design,  build,  test, 
identify  deficiencies,  fix,  retest,  and  repeat.  Development  Test  and 
Evaluation  (DT&E)  is  an  important  aspect  of  this  process.  The  DT&E  is 
performed  in  the  factory,  laboratory  and  on  the  proving  ground  by  the 
subcontractors  as  they  are  developing  the  components  and  subassembly, 
by  the  prime  contractor  as  he  assembles  the  components  and  ensures 
integration  of  the  system,  and  by  the  Government  to  demonstrate  how  well 
the  weapon  system  meets  its  technical  and  operational  requirements. 
This  module  describes  development  testing  and  the  various  types  of 
activities  it  involves.  The  module  also  discusses  how  development  testing 
is  used  to  support  the  technical  review  process. 


CHAPTER  7 

INTRODUCTION  TO  DEVELOPMENT  TEST  AND  EVALUATION 


7.1  INTRODUCTION 

Development  test  and  evaluation  is  that  test  and  evaluation 
conducted  to  demonstrate  that  the  engineering  design  and  development 
process  is  complete.  It  is  used  by  the  contractor  to  reduce  risk,  validate 
and  qualify  the  design  and  to  ensure  that  the  product  is  ready  for 
government  acceptance.  DT&E  results  are  evaluated  to  ensure  that  design 
risks  have  been  minimized,  that  the  system  will  meet  specifications, 
and  to  estimate  the  system's  military  utility  when  it  is  introduced  into 
service.  DT&E  also  serves  a  critical  purpose  in  reducing  the  risks  of 
development  by  testing  selected  high  risk  components  or  subsystems. 
Finally,  DT&E  is  the  government  Developing  Agency  (DA)  tool  to  confirm 
that  the  system  performs  as  technically  specified,  and  that  the  system 
is  ready  for  field  testing.  This  chapter  provides  a  general  discussion 
of  contractor  and  Government  DT&E  activities,  stresses  the  need  for  an 
Integrated  test  program,  describes  some  special  purpose  types  of  DT, 
and  discusses  several  factors  that  may  influence  the  extent  and  scope 
of  the  DT&E  program. 

7.2  DT&E  RESPONSIBILITIES 

As  illustrated  in  Figure  7-1,  the  primary  participants  in  testing 
are  the  prime  contractor,  subcontractor.  Service  materiel  developer  or 
developing  agency  and  the  operational  test  and  evaluation  agency.  In 
some  Services,  there  are  also  independent  evaluation  organizations  that 
assist  the  testing  organization  in  the  design  and  evaluation  of  development 
tests.  As  the  figure  shows,  system  development  testing  is  performed 
principally  by  contractors  during  the  early  development  stages  of  the 
acquisition  cycle  and  government  test/evaluation  organizations  during 
the  later  phases. 

The  Army  testing  of  the  Advanced  Attack  Helicopter  illustrates 
the  type  of  development  testing  performed  by  contractors  and  the 
relationship  of  this  type  of  testing  to  government  DT&E  activities. 

During  the  contractor  competitive  Phase  I  testing 
of  the  Army's  Advance  Attack  Helicopter  (AAH), 
the  prime  and  subcontractor's  testing  included 
design  support  tests,  testing  of  individual 
components,  establishing  limited  fatigue  lives, 
and  bench  testing  of  dynamic  components  to 
demonstrate  sufficient  structural  integrity  for 
the  conduct  of  the  Army  competitive  flight  test 
program.  Complete  dynamic  system  testing  was 
conducted  utilizing  ground  test  vehicles.  In 
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Source:  Adapted  &  Modified  from,  "Solving  the  Risk  Equation  In  Transiting 
From  Development  to  Production,"  January  19,  1984 


addition  to  supporting  the  contractor's  development 
effort,  these  tests  also  provided  Information 
for  the  Army's  technical  review  process  as  the 
Systems,  Preliminary  and  Critical  Design  Reviews 
were  conducted.  Following  successful  completion 
of  the  Ground  Tost  Vehicle  Qualification  Testing, 
first  flights  were  conducted  on  the  two  types 
of  competing  helicopters  with  each  aircraft  being 
flown  300  hours  before  delivery  of  two  of  each 
competing  aircraft  to  the  Army.  The  contractor 
flight  testing  was  oriented  toward  flight  envelope 
development,  demonstration  of  structural  Integrity, 
and  evaluation  and  verification  of  aircraft  flight 
handling  qualities.  Some  weapons  system  testing 
was  conducted  during  this  phase.  During  this 
phase,  government  testers  used  much  of  the 
contractor's  testing  data  to  develop  the  test 
data  matrix  as  part  of  the  government's  DT  and 
0T  planning  efforts.  The  use  of  contractor's 
test  data  reduced  the  testing  required  by  the 
Government,  and  added  validity  to  the  systems 
test  already  conducted  and  data  received  from 
other  sources. 


7.2.1  Contractor  Testing 

Materiel  development,  testing,  and  evaluation  is  an  iterative 
process  In  which  a  contractor  designs  hardware  and  software,  evaluates 
its  performance,  makes  changes  as  necessary  and  retests  for  performance 
and  technical  compliance.  Contractor  testing  plays  a  primary  role  in 
the  total  test  program,  and  the  results  of  contractor  tests  are  useful 
to  the  Government  evaluator  in  supporting  Government  test  objectives. 
It  is  important  that  Government  evaluators  oversee,  as  appropiate, 
contractor  system  tests  and  use  test  data  from  them  to  address  the  issues 
for  Government  testing.  It  is  not  uncommon  for  contractor  testing  to 
be  conducted  at  Government  test  facilities,  since  contractors  often  do 
not  have  the  required  specialized  facilities  (e.g.,  for  testing  hazardous 
components  or  for  missile  flight  tests).  This  enables  Government  evaluators 
to  monitor  the  tests  more  readily  and  increases  Government  confidence 
in  the  test  results. 

Normally,  a  Request  For  Proposal  (RFP)  requires  that  the  winning 
contractor  submit  an  Engineering  Design  Test  Plan  within  sixty  to  ninety 
days  after  contract  initiation  for  coordination  with  Government  test 
agencies  and  approval.  When  approved,  the  contractor's  test  program 
automatically  becomes  part  of  the  Development  Agency's  Integrated  Test 
Plan. 
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If  the  contractor  has  misinterpreted  the  RFP  requirements  and 
the  Engineering  Design  Test  Plan  does  not  satisfy  the  Government  test 
objectives,  the  Iterative  process  of  amending  the  contractor's  test  program 
begins.  This  Iterative  process  must  be  accomplished  within  limited  bounds 
so  that  the  contractor  can  meet  the  test  objectives  without  significant 
effects  on  contract  cost,  schedule,  or  scope. 

7.2.2  Government  Testing 

Government  testing  Is  performed  to  demonstrate  how  well  the 
materiel  system  meets  its  technical  compliance  requirements;  to  provide 
data  to  assess  developmental  risk  for  decision  making;  to  verify  that 
the  technical,  and  support  problems  Identified  In  previous  testing  have 
been  corrected;  and  to  ensure  that  all  critical  Issues  to  be  resolved 
by  testing  have  been  adequately  considered.  All  previous  testing  is 
considered  during  the  Government  evaluation  from  the  contractor's  bench 
testing  through  Development  Agency  testing  of  production  representative 
prototypes. 

Government  materiel  development  organizations  Include  major 
materiel  acquisition  commands  and,  in  some  cases,  operational  commands. 
The  materiel  acquisition  commands  have  test  and  evaluation  organizations 
that  conduct  government  development  testing.  In  addition  to  monitoring 
and  participating  In  contractor  testing,  these  organizations  conduct 
development  testing  on  selected  high  concern  areas  to  evaluate  the  adequacy 
of  systems  engineering,  design,  development  and  performance  to 
specifications.  The  Program  Management  Office  must  be  Involved  in  all 
stages  of  the  testing  these  organizations  perform. 

In  turn,  the  Materiel  Development/Test  and  Evaluation  Agencies 
conduct  test  and  evaluation  of  the  systems  in  the  development  stage  to 
ensure  that  they  meet  technical  and  operational  requirements.  These 
organizations  operate  Government  proving  grounds,  test  facilities  and 
labs  and  must  be  responsive  to  the  needs  of  the  Program  Manager  by  providing 
test  facilities,  personnel,  and  data  acquisition  services  as  required. 

7.2.3  Program  Manager's  Role 

The  Program  Manager  is  responsible  for  coordinating  the  test 
and  evaluation  program.  He  performs  this  task  with  the  assistance  of 
the  test  and  evaluation  working  group  whose  members  are  assembled  from 
development  agency  and  combat  development,  technical  and  operational  test 
and  evaluation,  logistics,  and  training  organizations.  The  Program  Manager 
must  remain  active  In  all  aspects  of  testing  including  planning,  funding, 
resourcing,  execution  and  reporting.  He  plays  an  Important  role  as  the 
Interface  between  the  contractor  and  the  government  testing  community. 
Recent  emphasis  on  early  T&E  highlights  a  need  for  early  government  tester 
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involvement  in  contractor  testing.  For  example,  during  development  of 
the  Army  Advanced  Attack  Helicopter  (AAH)  test,  it  was  found  that  having 
Program  Management  personnel  on  the  test  sites  improved  test  continuity, 
facilitated  the  flow  of  spares  and  repair  parts,  provided  a  method  of 
monitoring  the  contractor's  performance,  and  kept  the  Services  informed 
with  timely  status  reports. 

7.3  TEST  PROGRAM  INTEGRATION 

During  the  development  of  a  weapon  system,  there  are  a  number 
of  tests  that  are  conducted  by  subcontractors,  the  prime  contractor,  and 
the  Government.  To  ensure  that  these  tests  are  properly  time  phased, 
that  adequate  resources  are  available,  and  to  minimize  unneccessary  testing, 
a  coordinated  test  program  must  be  developed  and  followed.  The  Test  and 
Evaluation  Master  Plan  (TEMP)  normally  addresses  government-conducted 
tests  only;  it  does  not  provide  a  sufficient  level  of  detail  concerning 
contractor  or  subcontractor  tests.  A  contractor  or  PMO  Integrated  Test 
Plan  must  also  be  developed  to  describe  these  tests. 

7.3.1  Integrated  Test  Plan 

The  Integrated  Test  Plan  (ITP)  is  used  to  record  the  individual 
test  plans  for  the  subcontractor,  prime  contractor,  and  Government.  The 
prime  contractor  should  be  contractually  responsible  for  the  preparation 
and  updating  of  the  ITP,  and  the  contractor  and  Service  Developing  Agency 
should  ensure  that  it  remains  current.  The  ITP  includes  all  developmental 
tests  which  will  be  performed  by  the  prime  contractor  and  the  subcontractors 
at  both  the  system  and  subsystems  levels.  It  is  a  detailed  working-level 
document  which  assists  in  identifying  risk,  as  well  as  duplicative  or 
missing  test  activities.  A  well-maintained  ITP  facilitates  the  most 
efficient  use  of  test  resources. 

7.3.2  Single  Integrated  Test  Policy 

Most  Services  have  adopted  a  single  integrated  test  policy, 
thereby  reducing  much  of  the  government  testing  requirements.  This  policy 
stresses  independent  government  evaluation,  permits  an  evaluator  to  monitor 
both  contractor  and  government  test  programs,  and  evaluate  the  system 
from  an  independent  perspective.  The  policy  stresses  the  use  of  all 
available  test  data  for  system  evaluation. 

7.4  AREAS  OF  DTAE  FOCUS 
7.4.1  Life  Testing 

Life  Testing  is  performed  to  assess  the  effects  of  long-term 
exposure  to  various  portions  of  the  anticipated  environment  .  These  tests 
are  used  to  ensure  that  the  system  will  not  fail  prematurely  due  to  metal 
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fatigue,  component  aging,  or  other  problems  caused  by  long  term  exposure 
to  environmental  effects.  It  is  important  that  the  requirements  for  life 
testing  are  identified  early  and  integrated  into  the  system  test  plan. 
Life  tests  are  time  consuming  and  costly;  therefore,  life  testing 
requirements  and  life  characteristics  must  be  carefully  analyzed  concurrent 
with  the  initial  test  design.  Aging  failure  data  must  be  collected  early 
and  analyzed  throughout  the  testing  cycle.  If  life  characteristics  are 
ignored  until  results  of  the  test  are  available,  extensive  redesign  and 
project  delays  may  be  required.  Accelerated  life  testing  techniques  are 
available  and  may  be  used  whenever  applicable. 

7.4.2  Design  Evaluation/Verification  Testing 

Design  evaluation  and  verification  testing  is  conducted  by  the 
contractor  and/or  the  development  agency  with  the  primary  objective  of 
influencing  system  design.  Design  evaluation  is  fully  integrated  into 
the  development  test  cycle  and  its  purposes  are  to: 

(1)  Determine  if  critical  system  technical  characteristics 
are  achievable; 

(2)  Provide  data  for  refining  and  making  the  hardware  more 
rugged  so  it  will  comply  with  technical  specification 
requirements; 

(3)  Eliminate  as  many  technical  and  design  risks  as  possible 
or  to  determine  the  extent  to  which  they  are  manageable; 

(4)  Provide  for  evolution  of  design  and  verification  of  the 
adequacy  of  design  changes; 

(5)  Provide  information  in  support  of  development  efforts; 
and 

(6)  Ensure  components,  subsystems,  and  systems  are  adequately 
developed  before  beginning  Operational  Test  (OT). 

7.4.3  Design  Limit  Testing 

Design  limit  tests  are  integrated  into  the  test  program  to  ensure 
that  the  system  will  provide  adequate  performance  when  operated  at  the 
outer  performance  limits  and  when  exposed  to  environmental  conditions 
expected  at  the  extreme  of  the  operating  envelope.  The  tests  are  based 
on  mission  profile  data.  Care  must  be  taken  to  ensure  that  all  systems 
and  subsystems  are  exposed  to  the  worst  case  environments  with  adjustments 
made  because  of  stress  amplication  factors  and  cooling  problems.  Care 
must  also  be  taken  to  ensure  that  the  system  is  not  operated  beyond  the 
specified  design  limits.  For  example,  an  aircraft  component  may  have  to 
be  tested  at  temperature  extremes  from  an  artic  environment  to  a  desert 
environment. 
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7.4.4  Reliability  Development  Testing 

Reliability  Development  Testing  (RDT)  is  a  planned  Test,  Analyze, 
and  Fix  (TAAF)  process  in  which  development  items  are  tested  under  actual 
or  simulated  mission  profile  environments  to  disclose  design  deficiencies 
and  to  provide  engineering  information  on  failure  modes  and  mechanisms. 
The  purpose  of  RDT  is  to  provide  a  basis  for  early  incorporation  of 
corrective  actions  and  verification  of  their  effectiveness  in  improving 
the  reliability  of  equipment.  RDT  is  conducted  under  controlled  conditions 
with  simulated  operational  mission  and  environmental  profiles  to  determine 
design  and  manufacturing  process  weaknesses.  The  RDT  process  emphasizes 
reliability  growth  rather  than  a  numerical  measurement.  Reliability  growth 
during  RDT  is  the  result  of  an  Iterative  design  process  because  as  the 
failures  occur,  the  problems  are  identified,  solutions  proposed,  the 
redesign  is  accomplished  and  the  RDT  continues.  A  substantial  reliability 
growth  TAAF  testing  effort  was  conducted  on  the  F- 18  DT&E  for  selected 
avionics  and  mechanical  systems.  Although  the  TAAF  effort  added  $100 
million  to  the  RDT&E  program,  it  is  estimated  that  many  times  that  amount 
will  be  saved  through  lower  operational  costs  throughout  the  system's 
life. 

7.4.5  Reliability,  Availability  and  Maintainability  Assessment 

Reliability,  Availability  and  Maintainability  (RAM)  requirement* 
are  assessed  during  all  contractor  and  Government  testing.  Data  is 
collected  from  each  test  event  and  placed  in  a  RAM  data  base  which  is 
managed  by  the  development  agency.  Contractor  and  Government  development 
tests  provide  a  measure  of  the  system  RAM  against  stated  specifications 
in  a  controlled  environment.  The  primary  emphasis  of  RAM  collection  during 
the  DT  is  to  provide  an  assessment  of  the  system's  RAM  growth  and  a  basis 
for  assessing  the  consequences  of  any  differences  anticipated  during  field 
operations. 

7.5  SYSTEM  DESIGN  FOR  TESTING 

Built-in  Test  (BIT)  and  production  testing  are  two  major  areas 
that  must  be  considered  from  the  start  of  the  design  effort.  Design  for 
testing  addresses  the  need  to:  (1)  collect  data  during  the  development 
process  concerning  particular  performance  characteristics;  (2)  enable 
efficient  and  economic  production  by  providing  ready  access  to  and 
measurement  of  appropriate  acceptance  parameters;  and  (3)  enable  rapid 
and  accurate  assessment  of  the  status  of  the  product  to  the  lowest 
repairable  element  when  deployed.  Many  hardware  systems  have  testing 
circuits  designed  and  built-in.  This  early  planning  by  design  engineers 
allows  easy  test  for  fault  isolation  of  circuits,  both  in  development 
phases  of  the  system  development  and  during  operational  testing  and 
deployment.  There  are  computer  chips  in  which  more  than  one  half  of  the 
circuits  are  for  test/circuit  check  functions.  This  type  of  circuit  design 
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requires  early  planning  by  the  PM  to  ensure  that  the  RFP  Includes  the 
requirement  for  deslgned/bullt-in  test  capability. 

7.6  IMPACT  OF  WARRANTIES  ON  T&E 

A  warranty  or  guarantee  is  a  commitment  provided  by  a  supplier 
to  deliver  a  product  that  meets  specified  standards  for  a  specified  period 
of  time.  With  a  properly  structured  warranty,  the  contractor  is  required 
to  meet  technical  and  operational  requirements.  If  the  product  should 
fail  during  the  period  of  that  warranty,  the  contractor  must  replace  or 
make  repairs  at  no  additional  cost  to  the  Government.  The  Defense 
Appropriations  Act  of  1984  requires  warranties  or  guarantees  on  all 
procurements  of  weapon  systems.  This  act  makes  warranties  a  standard 
item  on  most  fixed  price  production  contracts.  Incentives  are  the  main 
thrust  of  warranties,  and  as  prescribed  in  MIL-STD-781,  the  Government 
will  perform  a  reliability  demonstration  test  on  the  system  to  determine 
these  incentives.  Although  warranties  have  favorable  advantages  to  the 
Government  during  the  early  years  of  the  contract,  warranties  do  not  affect 
the  types  of  testing  performed  to  ensure  that  the  system  meets  technical 
specifications  and  its  operational  effectiveness  and  suitability. 
Warranties  do,  howeve.r,  have  an  effect  on  the  amount  of  testing  required 
to  establish  reliability.  Because  the  standard  item  is  warranted,  less 
emphasis  on  that  portion  of  the  item  can  allow  for  additional  emphasis 
on  other  aspects  of  the  item  that  are  not  covered  under  the  warranty. 
Further,  the  Government  may  tend  to  have  more  confidence  in  the  contractor 
test  results  and  may  be  able,  therefore,  to  avoid  some  duplication  of 
test  effort.  The  warranty  essentially  shifts  the  burden  of  performance 
from  the  Government  to  the  contractor.  Warranties  can  increase 
significantly  the  price  of  the  contract,  especially  if  high  risk  components 
are  involved. 

7.7  DT&E  OF  LIMITED  PROCUREMENT  QUANTITY  PROGRAMS 

Programs  that  involve  the  procurement  of  relatively  few  items, 
typically  over  an  extended  time  period,  are  normally  subjected  to  standard 
DT&E.  Occasionally,  a  unique  test  approach  will  be  used  which  deviates 
from  the  standard  timing  and  reporting  schedule.  The  principle  of  DT&E 
of  components,  subsystems,  prototypes,  and  first  production  models  of 
the  system  is  normally  applied  to  limited  procurements.  It  is  important 
that  DT&E  and  OT&E  organizations  work  together  to  ensure  that  T&E  plans 
are  integrated  into  the  overall  acquisition  strategy. 

7.8  SUMMARY 

Development  Test  and  Evaluation  Is  an  iterative  process  of  design, 
build,  test,  identify  deficiencies,  fix,  retest,  and  repeat.  It  is 
performed  in  the  factory,  laboratory  and  on  the  proving  ground  by  the 
contractors  and  the  Government.  Contractor  and  Government  testing  is 
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combined  Into  one  Integrated  program  test  and  conducted  to  determine  if 
the  technical  development  of  the  acquisition  process  have  been  met,  and 
to  provide  data  to  the  decision  authority. 


CHAPTER  8 


DT&E  SUPPORT  OF  TECHNICAL  REVIEWS 
AND  MILESTONE  DECISIONS 


8.1  INTRODUCTION 

Throughout  the  acquisition  process,  development  test  and 
evaluation  is  oriented  toward  the  demonstration  of  the  completeness  and 
adequacy  of  systems  engineering,  design,  development  and  performance  to 
specifications.  DT&E  serves  a  critical  purpose  in  reducing  the  risks 
of  development  by  testing  and  evaluating  selected  high  risk  components 
or  subsystems.  DT&E  is  the  developer's  tool  to  establish  the  system 
performs  as  specified,  deficiencies  are  corrected  and  the  system  is  ready 
for  operational  testing  and  fielding.  DT&E  results  are  used  throughout 
the  system  engineering  process  to  provide  valuable  data  in  support  of 
formal  design  reviews.  This  chapter  describes  the  types  of  development 
testing  that  are  used  throughout  the  system  acquisition  cycle  to  support 
the  materiel  acquisition  process.  It  also  describes  the  objectives  of 
the  various  tests  conducted  during  the  DT&E  process  and  discusses  their 
relationship  to  the  formal  design  reviews  that  are  essential  to  the  systems 
engineering  process. 

8.2  DT&E  AND  THE  SYSTEM  ACQUISITION  CYCLE 

As  illustrated  in  Figure  8-1,  development  test  and  evaluation 
is  conducted  throughout  the  system  life  cycle.  DT&E  begins  before  program 
initiation  (milestone  0)  with  the  evaluation  of  evolving  technology,  and 
continues  after  the  system  is  in  service,  (prior  to  milestone  V). 

8.2.1  DT&E  Prior  to  Program  Initiation 

Prior  to  program  initiation,  technology  feasibility  testing 
is  conducted  to  confirm  that  the  technology  considered  for  the  proposed 
weapon  development  is  the  most  advanced  available  and  that  it  is  technically 
feasible. 


8.2.2  DT&E  During  the  Concept  Exploration/Definition  Phase 

Development  testing  that  takes  place  during  the  Concept 
Exploration/Definition  Phase  is  conducted  by  the  contractor  and  Government 
to  assist  in  selecting  preferred  alternative  system  concepts,  technologies, 
and  designs.  The  testing  conducted  depends  on  the  state  of  development 
of  the  test  article's  design.  Government  test  evaluators  participate 
in  this  testing  because  information  obtained  can  be  used  to  support  the 
Systems  Requirements  Review.  The  information  obtained  from  these  tests 
may  also  be  used  to  support  a  concept  selection  decision  by  the  Services 
or  OSD. 
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on  Process 


8.2.3  DT&E  During  the  Concept  Demonstration/Validation  Phase 

Development  testing  conducted  during  the  Concept 
Demonstration/Validation  Phase  is  used  to  demonstrate:  that  ail  technical 
risk  areas  have  been  identified  and  reduced  to  acceptable  levels;  that 
the  best  technical  approaches  have  been  accepted;  and  that  from  this  point 
on,  engineering  efforts,  rather  than  experimental  efforts,  will  be  required. 
It  supports  the  Milestone  II  decision  which  considers  entry  into  Full-Scale 
Development  and  as  appropriate  low  rate  initial  production.  This  DT&E 
includes:  contractor/government  integrated  testing,  engineering  design 
testing,  and  advanced  development  verification  testing. 

Development  testing  done  during  this  period  is  most  often  conducted 
at  the  contractor's  facility.  It  is  conducted  on  components,  subsystems, 
brassboard  configurations  or  advanced  development  prototypes  to  evaluate 
the  potential  application  of  technology  and  related  design  approaches 
prior  to  Full-Scale  Development.  Component  interface  problems  and  equipment 
performance  capability  are  evaluated.  The  use  of  properly  validated 
analysis,  modeling,  and  simulation  is  encouraged,  especially  during  the 
early  phases  to  assess  those  areas  that,  for  safety  or  testing  capability 
limitations,  cannot  be  directly  observed  through  testing.  Models  can 
provide  early  projections  of  systems  performance,  effectiveness  and 
suitability  and  can  reduce  testing  costs.  This  test  and  evaluation  also 
includes  an  initial  environmental  assessment. 

The  Army's  testing  of  the  Advanced  Attack  Helicopter  provides 
an  example  of  the  type  of  activities  that  occur  during  DT.  The  early 
DT&E  on  the  Advanced  Attack  Helicopter  (AAH)  was  conducted  by  the  Army 
Engineering  Flight  Activity.  The  test  was  conducted  in  conjunction  with 
an  Early  Operational  Test,  and  candidate  designs  were  flown  more  than 
90  hours  to  evaluate  flight  handling  qualities  and  aircraft  performance. 
This  test  also  included  the  firing  of  the  30  millimeter  cannon  and  the 
2.75  inch  rockets.  RAM  data  was  obtained  throughout  the  test  program 
and  this  data,  along  with  RAM  data  provided  from  early  contractor  testing, 
became  a  part  of  the  system's  RAM  database.  After  evaluating  the  results, 
the  Army  selected  a  contractor  to  proceed  with  Full-Scale  Development 
of  the  AAH. 

8.2.4  DT&E  During  the  Full-Scale  Development  Phase 

DT&E  conducted  during  the  Full-Scale  Development  Phase  provides 
the  final  technical  data  for  determining  a  system's  readiness  to  transition 
into  either  low-rate  initial  production  or  full-rate  production.  It  is 
conducted  using  prototype  hardware  and  is  characterized  by  the  use  of 
engineering  and  scientific  approaches  under  controlled  conditions.  The 
test  provides  quantitative  and  qualitative  data  for  use  in  the  system's 
evaluation.  The  evaluation  results  are  used  by  the  development  community 
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and  are  also  provided  to  Service  and  OSD  decision  authorities.  This  test 
measures  technical  performance  including:  effectiveness,  reliability, 
availability,  maintainabil ity,  compatibility,  interoperability,  safety, 
and  supportabil ity.  It  includes  tests  of  human  engineering  and  technical 
aspects  of  the  system  and  demonstrates  whether  engineering  is  reasonably 
complete  and  whether  solutions  to  all  significant  design  problems  are 
in  hand. 


As  an  example  of  testing  done  during  the  full-scale  development 
phase  the  Army  Advanced  Attack  Helicopter  was  flown  in  a  series  of 
Engineering  Design  Tests  (EDT).  The  EDT-1,  2,  and  4  were  flown  at  the 
contractor's  facility.  (The  EDT-3  requirement  was  deleted  during  a  program 
restructuring).  The  objectives  of  these  flight  tests  were  to  evaluate 
the  handling  characteristics  of  the  aircraft,  check  significant  performance 
parameters  and  confirm  the  correction  of  deficiencies  noted  during  earlier 
testing.  The  EDT-5  was  conduc ted  at  the  Army's  test  facility,  Yuma  Proving 
Ground..  The  objectives  of  this  test  were  the  same  as  earlier  EDTs;  however, 
the  testers  were  required  to  ensure  that  all  discrepancies  were  resolved 
prior  to  the  aircraft  going  into  operational  testing.  During  the  EDT's, 
operational  test  personnel  were  completing  Operational  Test,  test  design, 
bringing  together  test  resources  and  observing  the  DT&E  tests. 
Additionally,  OT  personnel  were  compiling  test  data  from  other  sources, 
such  as  the  system  contractor's  test  results.  The  evolving  DTtest  results, 
along  with  contractor  data,  were  made  available  to  the  Critical  Design 
Review  members  to  ensure  that  each  configuration  item  design  was  essentially 
completed.  Additionally,  a  Physical  Configuration  Audit  was  conducted 
by  the  Army  to  provide  a  technical  examination  to  verify  that  each  item 
"as  built"  conformed  to  the  technical  documentation  which  defined  that 
item. 

8.2.5  DT4E  During  the  Full-Rate  Production/Deployment  Phase 

Development  testing  may  be  conducted  throughout  the  LRIP  phase 
or  after  the  full-rate  production  decision  is  made  at  Milestone  III. 
This  test  is  normally  tailored  to  identify  design  problems  and  demonstrate 
the  system's  readiness  for  production.  This  testing  is  conducted  under 
controlled  conditions  and  provides  quantitative  and  qualitative  data. 
The  test  primarily  addresses  technical  performance  problems  and  the 
"ilities":  Reliability,  Maintainability,  Survivability,  and  Availability. 
This  testing  is  conducted  on  production  prototypes  of  production  items 
delivered  from  either  the  pilot  or  initial  production  runs.  It  is  conducted 
to  verify  the  system's  adequacy  and  quality  when  it  is  produced  in  quantity. 
The  test  ensures  that  the  items  are  produced  according  to  contract 
specification,  using  quantity  production  processes.  This  test  determines 
whether  the  system  is  successfully  transitioning  from  engineering 
development  prototype  to  production  and  that  the  system  meets  design 
specifications.  The  testing  performed  during  this  phase  includes  testing 
to  confirm  corrections  made  as  a  result  of  the  evaluation  of  problems 
disclosed  during  previous  development  testing. 
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8.2.6  Post  Production  T4E 

The  development  testing  that  occurs  soon  after  the  initial 
Operating  Capability  or  initial  deployment  assesses  the  deployed  system's 
operational  readiness  and  supportability.  It  ensures  that  all  of  the 
deficiencies  noted  during  previous  testing  have  been  corrected,  evaluates 
proposed  product  improvements  and  block  upgrades,  and  ensures  that 
integrated  logistics  support  is  complete.  It  also  evaluates  the  resources 
on  hand  and  the  plans  to  ensure  operational  phase  readiness  and  support 
objectives  are  sufficient  to  achieve  and  maintain  the  system  for  the 
remainder  of  its  acquisition  life  cycle.  Near  the  end  of  the  system's 
life,  DT&E  is  performed  to  assist  in  modification  of  the  system  to  help 
meet  new  threats  of  technologies  and  to  aid  in  disposal. 

Once  a  system  approaches  the  end  of  its  usefulness,  the 
development  testing  conducted  at  that  stage  is  concerned  with  the  monitoring 
of  a  system's  current  state  of  operational  effectiveness,  suitability, 
and  readiness  to  determine  whether  major  upgrades  are  necessary  or 
deficiencies  warrant  consideration  of  total  system  replacement.  Tests 
are  normally  conducted  by  the  Operational  Testing  community;  however, 
the  T&E  community  may  be  required  to  assess  the  technical  aspects  of  the 
system. 

8.3  DTSE  AND  THE  REVIEW  PROCESS 

8.3.1  The  Technical  Review  Process 

Technical  reviews  and  audits  are  conducted  by  the  Government 
and  the  contractor  as  part  of  the  system  engineering  process  to  ensure 
the  design  meets  the  system,  subsystem  and  software  specifications.  Each 
review  is  unique  in  its  timing  and  its  orientation.  Some  reviews  build 
on  previous  reviews  and  take  the  design  and  testing  effort  one  step  closer 
to  the  final  system  design  to  satisfy  the  operational  concept/purpose 
for  the  weapon  system.  Figure  8-2  illustrates  the  sequencing  of  the 
technical  reviews  in  relation  to  the  test  and  evaluation  phases. 

The  review  process  was  established  to  ensure  that  the  system 
under  development  would  meet  the  government's  requirements.  The  reviews 
evaluate  data  from  contractor  and  government  testing,  engineering  analysis, 
and  models  to  determine  if  the  system  or  its  components  will  eventually 
have  the  ability  to  meet  all  functional  and  physical  specifications  and 
to  determine  the  final  system's  design.  The  system  specification  is  very 
important  in  this  process.  It  is  the  document  used  as  a  benchmark  to 
compare  contractor  progress  in  designing  and  developing  the  desired  product. 
The  requirements  and  direction  for  these  formal  technical  reviews  and 
audits  are  set  forth  in  MIL-STD-1521B. 
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8.3.2  Testing  in  Support  of  Technical  Reviews 

The  testing  community  must  be  continually  involved  in  the 
technical  reviews  of  their  systems.  Decisions  made  at  these  reviews  have 
major  impacts  on  the  system  test  design,  resources  required  to  test,  and 
the  development  of  the  TEMP  and  other  documentation.  A  more  detailed 
discussion  of  testing  to  support  the  Technical  Reviews  is  provided  in 
the  Systems  Engineering  Management  Guide  (Reference  45).  The  reviews 
focus  primarily  on  the  Government's  technical  specifications  for  the  system. 
Figure  8-3  illustrates  the  program  specifications  and  how  they  are  developed 
in  the  system  life  cycle. 

8.3.3  Formal  Reviews 

8. 3. 3.1  Systems  Requirements  Review  (SRR) 

The  SRR  is  normally  conducted  during  the  System  Concept 
Exploration  or  Demonstration/Validation  Phases.  It  consists  of  a  review 
of  the  system/system  segment  specifications,  also  known  as  the  "A" 
specifications  (System  Functional  Block  Diagram,  Reference  45,  Chapter 
12),  and  is  conducted  after  the  accomplishment  of  functional  analysis 
and  preliminary  requirements  allocation.  During  this  review,  the  systems 
engineering  management  activity  and  its  output  are  reviewed  for 
responsiveness  to  the  Statement  of  Work  requirements.  The  primary  function 
of  the  SRR  is  to  ensure  that  systems  requirements  have  been  completed 
and  properly  identified  and  that  there  is  a  mutual  understanding  between 
the  contractor  and  the  Government.  During  the  review,  the  contractor 
describes  his  progress  and  any  problems  in  risk  identification  and  ranking, 
risk  avoidance  and  reduction,  trade-off  analysis,  producibility  and 
manufacturing  considerations,  and  hazards  considerations.  The  results 

of  integrated  test  planning  are  reviewed  to  ensure  the  adequacy  of  planning 
to  assess  the  design  and  to  identify  risks. 

8. 3. 3. 2  Systems  Design  Review  (SDR) 

The  SDR  is  conducted  as  a  final  review  prior  to  submittal  of 

the  Concept  Demonstration/Validation  Phase  products  or  as  the  initial 
Full-Scale  Development  Review.  The  "A"  specification  is  validated  to 
ensure  that  the  most  current  specification  is  included  in  the  System 
Functional  Baseline  and  that  they  are  adequate  and  cost  effective  to  satisfy 
validated  mission  requirements.  The  SDR  encompasses  the  total  system 

requirement  of  operations,  maintenance,  test,  training,  computers, 
facilities,  personnel  and  logistics  considerations.  A  technical 
understanding  should  be  reached  on  the  validity  and  the  degree  of 

completeness  of  the  specifications,  design,  operational  concept 
documentation,  software  requirements  specifications  and  interface 
requirements  specifications  during  this  review. 
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Figure  8-3.  Specifications  Summary 
Source.  Systems  Engineering  Management  Guide 


8. 3. 3. 3  Software  Specification  Review  (SSR) 

The  SSR  is  a  formal  review  of  the  Computer  System  Configuration 
Item  (CSCI)  requirements,  normally  held  after  a  SDR  but  prior  to  the  start 
of  a  CSCI  preliminary  design.  Its  purpose  is  to  validate  the  allocated 
baseline  for  preliminary  CSCI  design  by  demonstrating  to  the  government 
the  adequacy  of  the  Software  Requirements  Specifications,  Interface 
Requirements  Specifications,  and  Operational  Concept  Documentation. 

8. 3. 3. 4  Preliminary  Design  Review  (PDR) 

The  PDR  is  a  formal  technical  review  of  the  basic  approach  for 
a  configuration  item.  It  is  conducted  at  both  the  configuragtion  item 
and  system  level  early  in  the  Full-Scale  Development  Phase  to  confirm 
that  the  preliminary  design  logically  follows  the  SDR  findings  and  meets 
the  system  requirements.  The  review  results  in  an  approval  to  begin  the 
detailed  design.  The  Draft  Type  B  Specifications  are  reviewed  during 
the  PDR.  The  purpose  of  the  PDR  is  to:  evaluate  the  progress,  technical 
adequacy,  and  risk  resolution  (on  technical,  cost,  and  schedule  basis) 
of  the  configuration  item  design  approach,  DT  &  OT  activities  to  support 
the  performance  of  each  Cl,  and  establish  the  existence  and  compatibility 
of  the  physical  and  functional  interface  among  the  Cl  and  other  equipment. 

8. 3. 3. 5  Critical  Design  Review  (CDR) 

The  CDR  may  be  conducted  on  each  configuration  item  and/or  at 
the  system  level.  It  is  conducted  during  the  Full-Scale  Development  Phase 
when  the  detailed  design  is  essentially  complete,  prior  to  the  Functional 
Configuration  Audit.  During  the  CDR,  the  overall  technical  program  risks 
associated  with  each  configuration  item  are  also  reviewed  on  a  technical, 
cost  and  schedule  basis.  It  includes  a  review  of  the  "C"  Specifications 
and  the  status  of  both  the  system's  hardware  and  software. 

8. 3. 4. 6  Test  Readiness  Review  (TRR) 

The  TRR  is  a  formal  review  of  the  contractor's  readiness  to 
begin  Computer  System  Configuration  Item  (CSCI)  testing.  A  Government 
witness  will  observe  the  system  demonstration  to  verify  that  the  system 
is  ready  to  proceed  with  CSCI  testing.  It  is  conducted  after  the  software 
test  procedures  are  available  and  Computer  Software  Components  testing 
is  complete.  The  purpose  of  the  TRR  is  for  the  PMO  to  determine  whether 
the  contractor  is,  in  fact,  ready  to  begin  CSCI  testing. 

8. 3.4.7  Functional  Configuration  Audit  (FCA) 

The  FCA  is  a  formal  review  to  verify  that  the  configuration 
item's  actual  performance  complied  with  its  development  specification. 
The  "B"  Specification  are  derived  from  the  system  requirements  and  baseline 
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documentation.  During  the  FCA,  all  relevant  test  data  is  reviewed  to 
verify  that  the  item  has  performed  as  required  by  its  functional  and/c r 
allocated  configuration  identification.  The  audit  is  conducted  on  that 
item  which  is  representative  (prototype  or  production)  of  the  configuration 
to  be  released  for  production.  The  audit  consists  of  a  review  of  the 
contractor's  test  procedures  and  results.  Information  provided  will  be 
used  by  the  FCA  to  determine  the  status  of  planned  tests. 

8. 3. 3. 8  Physical  Configuration  Audit  (PCA) 

The  PCA  is  a  formal  review  which  establishes  the  product  baseline 
as  reflected  in  an  early  production  configuration  item.  It  is  the 

examination  of  the  as-built  version  of  both  hardware  and  software 
configuration  items  against  its  technical  documentation  .  The  PCA  also 

determines  that  the  acceptance  testing  requirements  prescribed  by  the 
documentation  are  adequate  for  acceptance  of  production  units  of  a  Cl 

by  quality  assurance  activities.  It  includes  a  detailed  audit  of 
engineering  drawings,  final  Part  II  product  specifications,  technical 
data,  and  plans  for  testing  that  will  be  utilized  during  production. 
The  PCA  is  performed  on  all  first  articles  and  on  the  first  CIs  delivered 
by  a  new  contractor. 

8. 3. 3.9  Formal  Qualification  Review  (FQR) 

The  FQR  is  a  systems  level  configuration  audit  that  is  conducted 
after  IOC  and  to  support  the  Milestone  IV  decision.  The  objective  is 
to  verify  that  the  actual  performance  of  the  Cl  as  determined,  through 
test,  complies  with  its  Type  "B"  Specifications  and  to  document  the  results 
of  the  qualification  tests.  The  FQR  and  FCA  are  often  performed  at  the 
same  time;  however,  if  sufficient  test  results  are  not  available  at  the 
FCA  to  ensure  the  Cl  will  perform  in  its  operational  environment,  the 
FQR  can  be  scheduled. 

8.3.3.10  Production  Readiness  Review  (PRR) 

The  PRR  is  an  assessment  of  the  contractor's  ability  to  produce 
the  items  on  the  contract.  It  is  usually  a  series  of  reviews  conducted 
prior  to  an  LRIP  or  full-scale  production  decision.  For  more  information, 
see  chapter  15,  paragraph  15.3 

8.3.3.11  Configuration  Change  Control 

The  Configuration  Change  Control  review  is  an  assessment  of 
the  impact  of  engineering  or  design  changes.  It  is  conducted  by  the 
engineering  configuration  control,  T&E,  and  PM  portions  of  the  PMO.  Every 
engineering  change  proposal  will  require  additional  testing,  and  the  TEMP 
must  reflect  the  new  schedules  and  resource  requirements.  Adequate  testing 
must  be  accomplished  to  ensure  integration  and  compatibility  of  these 
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changes.  For  example,  an  engineering  change  review  was  conducted  to 
integrate  color  monitors  into  the  Airborne  Warning  and  Control  System 
(AWACS)  to  replace  the  black  and  white  monitors.  Further,  the  AWACS 
operating  software  had  to  be  upgraded  to  handle  color  enhancement.  The 
review  was  conducted  on  the  Government  Program  Management  Office  (PMO) 
and  then  sections  of  the  PMO  were  tasked  to  contract,  test,  engineer, 
logistical ly  support,  control,  cost,  and  finance  the  change  to  completion. 
Configuration  control  and  engineering  changes  are  discussed  in  MIL-STD-481A. 

8.4  SUMMARY 

Design  reviews  are  an  integral  and  essential  part  of  the  system 
engineering  process.  The  meetings  range  from  very  formal  reviews  by  the 
Government  and  contractor  Program  Managers  to  informal  technical  reviews 
concerned  with  product  or  task  elements  of  the  work  breakdown  structure. 
All  reviews  share  the  common  objective  of  determining  the  technical  adequacy 
of  the  existing  design  to  meet  technical  requirements.  The  DT/OT 
assessments  and  test  results  are  made  available  to  the  reviews,  and  it 
is  important  that  the  test  community  be  involved  in  the  completion  of 
all  the  open  action  items. 


8-11 


MODULE  III 


OPERATIONAL  /  USER  TESTING 


Operational  Test  and  Evaluation  (OT&E)  is  conducted  to  ensure  that  a 
weapon  system  meets  the  validated  requirements  of  the  user  in  a  realistic 
scenario.  Operational  tests  are  focused  on  operational  requirements, 
effectiveness  and  suitability,  and  not  the  proof  of  engineering  specifi¬ 
cations,  as  is  the  case  with  development  testing.  This  module  provides  an 
overview  of  OT&E  and  discusses  how  OT&E  results  provide  essential  infor¬ 
mation  for  milestone  decisions. 


CHAPTER  9 

INTRODUCTION  TO  OPERATIONAL  TEST  AND  EVALUATION 

9.1  INTRODUCTION 


This  chapter  provides  an  introduction  to  the  concept  of 
operational  test  and  evaluation  (OT&E).  It  outlines  the  purpose  of  OT&E, 
discusses  the  primary  participants  in  the  OT&E  process,  describes  several 
types  of  OT&E,  and  includes  some  general  guidelines  for  the  successful 
planning,  execution,  and  reporting  of  OT&E  programs. 

9.2  PURPOSE  OF  OT&E 

Operational  test  and  evaluation  is  conducted  for  major  programs 
by  an  organization  that  is  independent  of  the  developing,  procuring,  and 
using  commands.  It  is  normally  conducted  in  phases,  each  of  which  are 
keyed  to  a  decision  review  in  the  materiel  acquisition  process.  It  is 
conducted  with  typical  user  operators,  crews,  or  units  in  realistic  and 
operational  environments.  The  OT&E  provides  the  decision  authority  with 
an  estimate  of: 

(1)  The  military  utility,  operational  effectiveness  and 
suitability  of  the  new  system; 

(2)  The  system's  desirability,  considering  systems  already 
available,  and  the  operational  benefits  or  burdens  associated 
with  the  new  system; 

(3)  The  need  for  modifications  to  the  new  system; 

(4)  The  adequacy  of  doctrine,  organizations,  operating 
techniques,  tactics,  and  training  for  employment  of  the 
system;  the  adequacy  of  maintenance  support  for  the  system; 
and  the  adequacy  of  the  system's  performance  in  the 
countermeasures  environment. 

9.3  TEST  PARTICIPANTS 

OT&E  of  major  systems  is  managed  by  an  independent  testing  agency 
which  each  military  Service  is  required  to  maintain.  It  is  accomplished 
under  conditions  as  operationally  realistic  as  possible.  Personnel 
operating,  maintaining,  and  supporting  the  system  during  OT&E  are  trained 
to  a  level  commensurate  with  that  of  personnel  who  will  actually  perform 
these  functions  under  peacetime  and  wartime  conditions.  Program  management 
office  personnel  and  test  coordinating  groups  also  play  important  parts 
in  the  overall  OT&E  process. 
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9.3.1  Program  Management  Office 

Even  though  operational  testing  is  performed  by  an  independent 
organization,  the  Program  Manager  (PM)  plays  an  important  role  in  its 

planning,  reporting,  and  funding.  He  must  coordinate  program  activities 
with  the  test  community,  especially  the  Operational  Test  Agencies.  He 
also  helps  ensure  that  testing  addresses  the  critical  issues  and  provides 
feedback  from  testing  activities  to  contractors. 

At  each  milestone  review,  the  Program  Manager  is  required  to 
brief  the  decision  authority  on  the  testing  planned  and  completed  on  the 
program.  It  is,  therefore,  important  that  the  Program  Management  Office 
(PMO)  personnel  have  a  good  understanding  of  the  test  program  objectives 
and  that  they  work  with  the  operational  test  community  to  ensure  that 

the  OT&E  is  well  planned  and  adequate  resources  are  available.  The  PMO 
should  involve  the  test  community  by  organizing  Test  Coordinating  Groups 
at  the  program  initiation  and  establishing  channels  of  communication  between 
the  PMO  and  the  key  test  organizations.  The  PMO  can  often  avoid 

misunderstandings  by  aggressively  monitoring  the  system  testing  and 

providing  up-to-date  information  to  key  personnel  in  OSD  and  the  Services. 
The  PMO  staff  should  keep  appropriate  members  of  the  test  community  well 
informed  concerning  system  problems  and  the  actions  undertaken  by  the 
PMO  to  correct  them. 

9.3.2  Test  Coordinating  Groups 

The  Army's  Test  Integration  Working  Group  (TIWG),  Navy's  T&E 
Coordinating  Group  (TECG),  and  Air  Force's  Test  Planning  Working  Group 
(TPWG)  are  chartered  by  their  respective  Service  to  coordinate  and  integrate 
the  planning  and  execution  of  the  T&E  program.  The  Army  and  Air  Force 
groups  are  chaired  by  a  representative  of  the  Program  Management  Office, 
often  the  Program  Manager.  The  Navy's  T&ECG  is  chaired  by  the  development 
coordinator.  The  members  of  these  groups  represent  the  user,  development 
and  operational  testing,  independent  evaluation,  logistics,  training, 
and  contractor  communities,  as  appropiate.  The  functions  of  the  groups 
are  to:  facilitate  the  use  of  testing  expertise,  instrumentation, 
facilities,  simulations,  and  models;  integrate  test  requirements;  accelerate 
the  TEMP  coordination  process;  resolve  cost  and  scheduling  problems;  and 
provide  a  forum  to  ensure  that  test  and  evaluation  of  the  system  is 
coordinated.  The  existence,  of  a  test  coordinating  group  does  not  alter 
the  responsibilities  of  any  command  or  headquarters  and,  in  the  event 
of  disagreement  within  a  group,  the  issue  is  resolved  through  the  normal 
command/staff  channels.  Within  the  Air  Force,  the  TPWG  may  help  to  prepare 
the  test  portions  of  the  request  for  proposal  and  related  contractual 
documentation,  and  evaluate  the  contractors'  proposals.  In  all  of  the 
Services,  the  groups  help  develop  the  TEMP. 
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9.3.3  Service  Operational  Test  Agencies 


The  operational  test  and  evaluation  organizations  should  become 
involved  early  in  the  system's  life  cycle,  usually  at  program  initiation, 
where  they  can  begin  to  develop  strategies  for  the  actual  conduct  of  the 
operational  and  evaluation  tests.  As  test  planning  continues,  a  detailed 
TEMP  is  developed  and  the  test  resources  are  identified  and  scheduled. 
During  the  early  stages,  the  OTAs  structure  an  OT&E  program,  consistent 
with  the  approved  acquisition  strategy,  for  the  system,  identifies  critical 
operational  test  issues,  and  assess  the  adequacy  of  candidate  systems. 
As  the  program  moves  into  advanced  planning,  OT&E  efforts  are  directed 
toward  becoming  familiar  with  the  system,  encouraging  interface  between 
the  user  and  developer  and  further  refining  the  critical  operational  issues. 
Each  Service  has  an  independent  organization  dedicated  to  planning, 
executing,  and  reporting  the  results  of  that  Service's  operational  test 
and  evaluation  activities.  These  organizations  include  the:  Army 
Operational  Test  and  Evaluation  Agency  (OTEA),  Navy  Operational  Test  and 
Evaluation  Force  (OPTEVFOR),  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC),  and  Marine  Corps  Operational  Test  and  Evaluation  Activity 
(MCOTEA).  These  organizations  are  discussed  further  in  Chapter  22. 

9.3.4  Test  Personnel 

Operational  testing  is  conducted  on  materiel  systems  with  typical 
user  players  in  as  realistic  an  operational  environment  as  possible. 
It  uses  personnel  with  the  same  military  occupational  specialties  as  those 
who  will  operate,  maintain,  and  support  the  system  when  it  is  fielded. 
Participating  troops  are  trained  in  the  systems  operation  based  on  the 
Service's  operational  mode  summary  and  mission  profiles.  Because  some 
operational  tests  consist  of  force-on-force  tests,  the  forces  opposing 
the  tested  system  must  also  be  trained  in  threat  tactics  and  doctrine. 
For  operational  testing  conducted  before  full -rate  production,  most  of 
the  training  on  the  system  is  conducted  by  the  system’s  contractor. 
Usually,  the  contractor  trains  the  school  cadre,  and  those  cadre  train 
the  other  troops.  As  the  system  enters  full-rate  production,  the  Service 
assume  training  responsibilities. 

9.4  TYPES  OF  OT&E 

Operational  Test  and  Evaluation  can  be  subdivided  into  two  phases: 
Operational  testing  performed  before  the  full-rate  production/deployment 
decision  (Pre-production  OT&E)  and  the  operational  testing  performed  after 
the  production  decision.  The  Pre-production  OT&E  includes  Operational 
Assessments,  Initial  Operational  Test  and  Evaluation  (IOT&E)  and  Follow-On 
Operational  Test  and  Evaluation  (FOT&E).  The  operational  assessments 
begin  very  early  in  the  program,  frequently  after  program  initiation, 
and  continue  until  the  system  is  certified  ready  for  the  independent 
operational  test  and  evaluation.  The  independent  operational  test  and 
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evaluation  is  conduct. d  just  prior  to  the  full-rate  production  deployment 
decision  and  continues  until  Initial  Operating  Capability  (IOC)  is  achieved. 
After  the  full-rate  production/deployment  all  subsequent  operational  testing 
is  referred  to  as  Follow-On  Operational  Test  and  Evaluation  (FOT&E). 
In  the  Air  Force,  if  no  research  and  development  funding  is  committed 
to  a  system.  Qualification  Operational  Test  and  Evaluation  may  be  performed 
in  lieu  of  IOT&E.  The  Navy  uses  the  term  "OPEVAL"  to  aefine  the  operational 
evaluation  to  predict  the  operational  effectiveness  and  operational 
suitability  and  the  production  readiness  of  a  system. 

9.4.1  Early  Operational  Test  and  Evaluation 

Early  Operational  Test  and  Evaluation  is  conducted  primarily 
to  forecast  and  evaluate  the  operational  effectiveness  and  suitability 
of  the  weapon  system  during  development.  Operational  assessments  are 
conducted  on  the  developing  system  until  the  development  agency  certifies 
the  prototype  is  ready  for  Initial  Operational  Test  and  Evaluation. 

9.4. 1.1  Operational  Assessments 

Operational  Assessments  begin  after  program  initiation  vhen 
the  Operational  Test  Agencies  (OTAs)  start  their  estimates  of  operational 
effectiveness  and  suitability.  The  OTA  uses  any  testing  results  and  data 
from  other  sources  during  an  evaluation.  These  data  are  evaluated  by 
the  OTA  from  an  operational  point  of  view.  As  the  program  matures,  these 
operational  assessments  are  conducted  on  prototypes  and  preproduction 
articles  until  the  system  is  fully  developed  and  certified  ready  for  its 
Initial  Operational  Test  and  Evaluation  (IOT&E)  or  OPEVAL  in  the  Navy. 

9.4. 1.2  Initial  Operational  Test  and  Evaluation  (or  OPEVAL) 

Initial  Operational  Test  and  Evaluation  is  the  final  dedicated 
phase  of  OT&E  preceding  a  full-rate  production  decision.  IOT&E  is  the 
one  of  the  final  examinations  that  entails  dedicated  operational  testing 
of  production-representative  test  articles  using  typical  operational 
personnel  in  as  realistic  a  scenario  as  possible.  IOT&E  is  conducted 
by  an  operational  test  and  evaluation  agency  independent  of  the  contractor. 
Program  Management  Office,  or  Developing  Agency.  DOD  Directive  5000.3 
defines  the  test  conditions  under  which  such  testing  must  be  conducted: 

Operational  testing  shall  be  accomplished  in  an 
environment  as  operationally  realistic  as  possible, 
including  threat  representative  hostile  forces.  Typical 
users  should  operate  and  maintain  the  system  under 
conditions  simulating  combat  stress  and  peacetime 
condi tions. 
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Further,  IOT&E  must  be  conducted  without  system  contractor  personnel 
participation  as  set  forth  in  Public  Law  99-661  by  Congress.  The  results 
from  this  test  are  evaluated  and  presented  to  the  decision  authority  prior 
to  the  decision  to  enter  full-rate  production  and  to  support  the  beyond-LRIP 
decision.  This  phase  of  OT&E  addresses  the  critical  issues  identified 
in  the  Decision  Coordinating  Paper  (DCP)  and  the  TEMP. 

9.4.2  Follow-On  Operational  Test  and  Evaluation 

Follow-On  Operational  Test  and  Evaluation  (FOT&E)  is  conducted 
after  the  Milestone  III  decision.  The  tests  are  conducted  in  a  realistic 
tactical  environment  similar  to  that  used  in  OT&E,  but  a  greater  number 
of  test  items  may  be  used.  Normally  FOT&E  is  conducted  using  production 
systems.  Specific  objectives  of  FOT&E  include  the  testing  of  modifications 
that  are  to  be  incorporated  into  production  systems,  the  completion  of 
any  deferred  or  incomplete  IOT&E,  and  assessment  of  operational  availability 
(Ao),  to  include  spares  support.  The  tests  are  also  used  to  test  the 
system  in  a  different  platform  application,  new  tactical  applications, 
or  against  new  threats. 

9.4.3  Qualification  Operational  Test  and  Evaluation 

Qualification  Operational  Test  and  Evaluation  (QOT&E)  may  be 
performed  by  the  major  command,  user,  or  operational  test  and  evaluation 
agency  and  is  conducted  on  minor  modifications  or  new  applications  of 
existing  equipment  when  no  research  and  development  funding  is  required. 
An  example  of  a  program  in  which  QOT&E  was  performed  by  the  Air  Force 
is  the  A- 10  Air-to-Air  Self  Defense  Program  where  the  mission  of  the  A- 10 
was  expanded  from  strictly  ground  support  to  include  an  air-to-air  defense 
role.  To  accomplish  this  the  A-10  aircraft  was  modified  with  off-the-shelf 
AIM-9,  air-to-air  missiles,  and  QOT&E  was  performed  on  the  system  to 
evaluate  its  operational  effectiveness  and  suitability. 

9.5  TEST  PLANNING 

Operational  test  planning  is  one  of  the  most  important  parts 
of  the  OT&E  process.  Proper  planning  facilitates  the  acquisition  of  data 
to  support  the  determination  of  the  weapon  system's  operational 
effectiveness  and  suitability.  Planning  must  be  pursued  in  a  "deliberate, 
comprehensive  and  structured  manner.  Careful  and  complete  planning  may 
not  guarantee  a  successful  test  program,  but  inadequate  planning  can  result 
in  significant  test  problems,  system  failure,  and  cost  overruns. 

Operational  test  planning  is  conducted  by  the  OTA  after  program  initiation 
prior  to  each  operational  test  phase. 
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Operational  planning  can  be  divided  into  three  phases: 
earlyplanning,  advanced  planning  and  detailed  planning.  Early  planning 
entails  critical  operational  issue  development,  determining  the  concept 
of  operation,  envisioning  the  operational  environment,  and  developing 
mission  scenarios  and  resource  requirements.  Advanced  planning  encompasses 
the  determination  of  the  purpose  and  scope  of  testing,  identification 
of  critical  issues,  development  of  test  objectives,  establishment  of  a 
test  approach  and  estimating  test  resource  requirements.  Detailed  planning 
involves  the  development  of  step-by-step  procedures  to  be  followed  as 
well  as  the  coordination  of  resource  requirements  necessary  to  carry  out 
OT&E. 

9.5.1  Critical  Operational  Testing  Issues 

One  of  the  purposes  of  OT&E  is  to  resolve  critical  operational 
issues  about  the  system.  The  first  step  in  an  OT&E  program  is  to  identify 
these  critical  issues,  some  of  which  are  implicit  in  the  definition  of 
OT&E.  For  example,  "How  well  does  the  system  perform  a  particular  aspect 
of  its  mission?"  and  "Can  the  system  be  supported  logistical ly  in  the 
field?"  Other  issues  arise  from  questions  being  asked  about  the  system's 
performance  or  how  it  will  affect  other  systems  with  which  it  must  operate. 
Critical  issues  provide  focus  and  direction  for  the  operational  test  and 
are  normally  expressed  in  the  form  of  questions.  Identifying  the  issues 
is  analogous  to  the  first  step  in  the  system's  engineering  process;  that 
is,  defining  the  problem.  When  critical  issues  are  properly  addressed, 
deficiencies  in  the  system  can  be  uncovered  and  corrected.  They  form 
the  basis  for  a  structured  technique  of  analysis  by  which  detailed 
subobjectives  or  measures  of  effectiveness  (MOE)  can  be  established. 
During  the  operational  test,  each  subobjective  is  addressed  by  an  actual 
test  measurement.  After  these  issues  are  identified,  the  evaluation  plans 
and  test  design  are  developed  for  test  execution. 

9.5.2  Test  Realism 

Realism  in  an  OT&E  program  includes  all  of  the  characteristics 
that  make  the  test  look,  sound,  feel,  taste,  and  smell  like  actual  combat 
operations.  In  order  to  achieve  realism  in  an  OT&E,  there  must  be  a  concern 
for  realism  throughout  the  planning  and  conduct  of  the  test.  The  three 
basic  areas  of  particular  significance  in  applying  detailed  considerations 
of  realism  identified  by  Roger  Smith  in  his  book,  "Operational  Test  and 
Evaluation:  A  Systems  Engineering  Approach",  are: 

(1)  During  development  of  the  test  concept  paper  and  design 
of  the  over  all  aspects  of  the  test  program,  the  developers 
must  ensure  the  basic  test  philosophy  is  determined  and 
realism  is  closely  woven  into  this  design. 

(2)  During  planning  and  design  of  the  actual  test  and  development 
of  scenarios,  the  planners  must  ensure  that  realism  is 
included  into  the  operational  and  maintenance  activities. 
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(3)  Durfng  the  actual  conduct  of  the  tests,  the  field  testers 
must  ensure  that  the  tactical  realism  is  maintained. 

9.5.3  Selection  of  a  Test  Concept 

An  important  step  in  the  development  of  an  OT&E  concept  is  to 
define  the  overall  test  program  concept.  It  must  be  determined  if  OT&E 
will  be  conducted  in  parallel  with  systems  development,  if  all  testing 
is  to  be  done  on  production  equipment,  if  testing  will  be  evolutionary, 
and  if  testing  will  have  to  wait  until  all  system  capabilities  are 
developed.  These  questions  can  best  be  answered  by  considering  a  number 
of  systems  aspects  such  as  test  information  requirements,  system 
availability  during  test  periods,  and  the  implementation  of  system 
capabilities.  The  test  concept  is  driven  by  the  acquisition  strategy 
and  is  a  road  map  used  for  test  design  (para  5.5.3)  and  evaluation. 

9.6  TEST  EXECUTION 

An  operational  test  plan  is  only  as  good  as  the  execution  of 
that  plan.  The  execution  is  the  essential  bridge  between  test  planning 
and  test  reporting.  The  test  is  executed  through  the  OT&E  test  director's 
efforts  and  the  actions  of  the  test  team.  For  successful  execution  of 
the  OT&E  plan,  the  test  director  must  direct  and  control  the  test  resources 
and  collect  the  data  required  for  the  evaluator  to  present  to  the  decision 
authority.  He  must  prepare  for  testing,  activate  and  train  the  test 
team,  develop  test  procedures  and  operating  instructions,  control  data 
management,  create  OT&E  plan  revisions,  and  manage  each  of  the  test 
missions.  His  data  management  duties  will  encompass  raw  data  collection, 
creating  a  data  status  matrix,  ensuring  data  quality  assurance,  processing 
and  reduction,  verification,  filing,  storage,  retrieval,  and  analysis. 
Once  all  the  tests  have  been  completed  and  the  data  is  reduced  and  analyzed, 
the  results  must  be  reported.  A  sample  test  organization,  the  Army 
organizational  framework  for  OT&E  of  the  I  81mm  Mortar,  is  illustrated 
in  Figure  9-1.  (In  the  Army,  the  Deputy  Test  Director  from  the  OTA  actually 
controls  the  operational  test  activity.) 

9.7  TEST  REPORTING 

The  OT&E  test  report  is  a  very  important  document.  It  must 
communicate  the  results  of  completed  tests  to  decision  authorities  in 
a  timely,  factual,  concise,  comprehensive  and  accurate  manner.  The  report 
must  present  a  balanced  view  of  the  weapon  system's  successes  and  failures 
during  testing,  and  illuminate  the  positive  aspects  and  the  system’s 
deficiencies  discovered. 

There  are  four  types  of  reports  most  frequently  used  in  reporting 
OT&E  results.  These  include  status,  interim,  quick-look,  and  final  reports. 
The  status  report  gives  periodic  updates  (e.g.,  monthly,  quarterly)  and 
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Figure  9-1.  Organizational  Breakdown  of  the  I81mm  Mortar  Operational  Test  Directorate 


reports  recent  test  findings  (discreet  events  such  as  missile  firings). 
The  interim  report  provides  a  summary  of  the  cumulative  test  results  to 
date.  The  quick-look  report  provides  preliminary  test  results,  are  usually 
prepared  Immediately  after  a  test  event  (less  than  7  days)  and  may  be 
used  to  support  program  decision  milestones  due  to  the  need  to  support 
a  decision  before  the  final  report  can  be  written.  The  final  test  report 
(Air  Force,  Navy),  or  Army  evaluation  report  presents  the  final  test  results, 
conclusions,  and  recommendations  covering  the  entire  OT&E  program  with 
all  supporting  data. 

9.8  SUMMARY 

The  purpose  of  operational  test  and  evaluation  is  to  assess 
operational  effectiveness  and  suitability  at  each  stage  in  the  acquisition 
process.  Operational  effectiveness  is  a  measure  of  the  contribution  of 
the  system  to  mission  accomplishment  under  actual  conditions  of  employment. 
Operational  suitability  is  a  measure  of  the  maintainability  and  reliability 
of  the  system,  the  effort  and  level  of  training  required  to  maintain, 
support,  and  operate  it,  and  any  unique  logistic  or  training  requirements 
it  may  have.  OT&E  may  also  provide  information  on  tactics,  doctrine, 
organization,  and  personnel  requirements  and  may  be  used  to  assist  in 
the  preparation  of  operating  and  maintenance  instructions  and  other 
publications.  Its  most  important  aspect  is  that  it  provides  an  independent 
evaluation  of  the  utility  of  the  system  and  the  feasibility  of  employing 
it. 
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CHAPTER  10 

OTAE  TO  SUPPORT  MILESTONE  DECISIONS 
10.1  INTRODUCTION 


Operational  test  and  evaluation  is  conducted  in  keeping  with 
principles  of  objectivity  and  an  impartial  evaluation  to  provide  to  the 
decision  authority,  prior  to  each  major  milestone  review,  the  operational 
information  necessary  to  resolve  critical  operational  issues.  The 
principles  of  testing  are  summarized  in  three  terms--adequacy,  quality, 
and  credibility: 

Adequacy--The  amount  of  data  and  realism  of  test  conditions 
must  be  sufficient  to  support  the  resolution  of  the  critical  issues. 

Quality--The  test  planning,  control  of  test  events,  and  treatment 
of  data  must  make  the  operational  information  clear  and  accurate. 

Credibility--The  conduct  of  the  test  and  data  handling  must 
be  separated  from  external  influence  and  personal  self-interest. 

This  chapter  discusses  the  operational  testing  conducted  to 
provide  information  to  support  DOD  executive  level  management  decisions 
on  major  acquisition  programs.  Figure  10-1  illustrates  how  T&E  relates 
to  the  acquisition  process. 

10.2  OTAE  DURING  THE  CONCEPT  EXPLORATION/DEFINITION  PHASE 

(Milestone  0  to  Milestone  I) 

Operational  Test  and  Evaluation  (OT&E)  is  accomplished  using 
a  test  cycle  of  successive  actions  and  documents.  During  the  early  stages 
of  the  program,  the  process  is  informal  and  modified  as  necessary.  As 
programs  mature,  documentation  for  major  systems  and  those  designated 
by  DOT&E  for  oversight  must  be  sent  to  OSD  for  approval  before  the  testing 
can  be  conducted  or  the  systems  can  be  cleared  to  proceed  into  low-rate 
initial  production.  OT&E  conducted  during  the  Concept 
Exploration/Definition  (CED)  Phase,  operational  assessment,  is  focused 
on  investigating  the  deficiencies  identified  during  the  mission  area 
analysis.  Operational  testers  participate  in  these  evaluations  to  validate 
the  OT&E  requirements  for  future  testing  and  to  identify  those  issues 
and  criteria  which  can  only  be  resolved  through  OT&E  in  order  to  initiate 
early  test  resource  planning. 

The  OT&E  objectives  prior  to  Milestone  I  are  to  assist  in 
selecting  alternatives  to  resolve  the  mission  area  deficiencies,  and  to 
assess  the  operational  impact  of  the  system.  This  early  assessment  also 
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provides  data  to  support  a  decision  on  whether  or  not  to  enter  the  Concept 
Demonstration/Validation  phase.  OT&E  conducted  during  the  CED  phase 
supports  the  development  of  estimates  of: 

(1)  The  military  need  for  the  proposed  system; 

(2)  A  demonstration  that  there  is  a  sound  physical  basis  for  a  new 
system; 

(3)  An  analysis  of  the  concept,  based  on  demonstrated  physical 
phenomena,  for  satisfying  the  military  need; 

(4)  The  system's  affordability  and  life-cycle  cost; 

(5)  The  ability  of  a  modification  to  an  existing  U.S.  or  Allied 
system  to  provide  needed  capability; 

(6)  An  operational  utility  assessment;  and 

(7)  An  impact  of  the  system  on  the  force  structure. 

At  Milestone  I,  there  is  normally  no  hardware  available  for 
the  operational  tester.  Therefore,  the  early  operational  assessment  is 
conducted  from:  surrogate  force  development  test  and  experiment  data, 
breadboard  models,  factory  user  trials,  mock-up/simulators,  and  user 
demonstrations.  This  makes  the  early  assessments  difficult  and  some  areas 
cannot  be  covered  in-depth;  however,  these  assessments  provide  vital 
introductory  information  on  the  systems  potential  operational  utility. 

The  OT&E  products  from  this  phase  of  testing  include  the 
information  provided  to  the  decision  authority,  data  collected  for  further 
evaluation  and  the  input  to  the  Decision  Coordinating  Paper  (DCP),  Test 
and  Evaluation  Master  Plan  (TEMP),  and  early  test  and  evaluation  plans. 
Special  logistics  problems,  program  objectives,  program  plans,  performance 
parameters,  areas  of  major  risk,  system  alternatives,  and  acquisition 
strategy  are  areas  of  primary  concern  to  the  operational  tester  during 
this  phase  and  must  be  carefully  evaluated  in  order  to  project  the  system 
operational  capabilities. 

10.3  OT&E  DURIN6  THE  CONCEPT  DEMONSTRATION/VALIDATION  PHASE 
(Milestone  I  to  Milestone  II) 

OT&E  during  the  Concept  Demonstration/Validation  Phase,  is 
conducted  to  support  the  Milestone  II  (MS- II)  decision  regarding  a  system's 
readiness  to  move  into  Full-Scale  Development.  In  all  cases,  appropriate 
and  adequate  T&E  must  be  conducted  prior  to  the  MS- I I  decision,  thereby 
reducing  risk  and  uncertainty  before  more  resources  are  committed.  As 
appropriate,  Low-Rate  Initial  Production  (LRIP)  of  selected  components 
and  quantities  may  be  approved  at  MS- I I  to  verify  production  capability 
and  to  provide  test  resources  needed  to  conduct  interoperability,  live 
fire,  or  operational  testing. 
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10.3.1  Objectives  of  Early  Operational  Asessnents 

Early  operational  assessment  efforts  are  conducted  to  Identify 
the  best  approach,  indicate  the  risks  and  solutions  for  this  phase  of 
the  development,  examine  operational  aspects  of  the  systems  development, 
and  estimate  potential  operational  effectiveness  and  suitability. 
Additionally,  an  analysis  of  the  planning  for  transition  from  development 
to  production  is  initiated.  The  Operational  Assessments  to  support  the 
MS- 1 1  decision  are  intended  to: 

(1)  Assess  the  potential  of  the  new  system  in  relation  to  existing 
capabilities. 

(2)  Assess  system  effectiveness  and  suitability  so  that  affordability 
can  be  evaluated  in  terms  of  program  cost  versus  the  military 
value. 

(3)  Assess  the  adequacy  of  the  concept  for  employment,  supportabil ity 
and  organization;  doctrinal,  tactical,  and  training  requirements; 
and  related  critical  issues. 

(4)  Estimate  the  need  for  the  selected  systems  in  consideration 

of  the  threat  and  system  alternatives  based  on  military  utility. 

(5)  Assess  the  validity  of  the  operational  concept. 

(6)  List  the  key  risk  areas  and  critical  operational  issues  that 

need  to  be  resolved  before  Full-Scale  Development  is  initiated. 

(7)  Assess  the  need  for  Low-Rate  Initial  Production  of  hardware 
to  support  testing  prior  to  the  full-rate  production  decision. 

The  OT&E  during  this  phase  may  be  conducted  on  brassboard 
configurations,  experimental  prototypes,  or  advanced  development  prototypes. 
There  may  also  be  an  advanced  prototype  system  available  for  the  operational 
tester.  However,  the  OT&E  assessments  may  also  make  use  of  many  other 
additional  data  sources.  Examples  of  additional  sources  often  used  by 
the  Army  during  this  phase  include:  Concept  Evaluation  Program  Tests, 

Innovative  Testing,  Force  Development  Tests  or  Experimentations  (FDT&E), 
Source  Selection  Tests,  User  Participation  in  DT&E,  and  Operational 
Feasibility  Tests.  The  results  from  this  testing,  analysis,  and  evaluation 
are  documented  in  the  DCP.  This  document  along  with  the  mission  needs 

documentation  and  Test  and  Evaluation  Master  Plan  assist  in  the  review 
process  for  the  MS-II  decision. 

10.3.2  OT-I  of  the  Advanced  Attack  Helicopter 

The  Army's  testing  of  the  Advanced  Attack  Helicopter  (AAH) 
provides  an  example  of  operational  testing  conducted  during  the 
Demonstration/Val idation  Phase: 

In  this  program,  DT-I  and  OT-I  activities  were 
integrated  wherever  possible.  The  OT-I  compared  the 
two  candidate  systems,  (YAH-63  and  YAH-64)  with  their 
respective  baseline  (each  an  AH- IS)  under  limited 
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operational  conditions  to  examine  the  following  aspects 
relative  to  mission  performance:  reliability, 
availability,  and  maintainability  characteristics; 
combat  survivability;  and  human  factors  data.  The 
tests  were  conducted  by  DT&E  pilots  at  Edwards  Air 
Force  Base  and  China  Lake  Naval  Weapons  Center, 
California.  The  aircraft  were  tested  in  the 
airframe-only  configuration  (i.e.,  without  weapons 
and  target  acquisition  subsystems),  for  16  hours  per 
airframe.  Within  this  limited  time,  sample  operational 
events  included:  hover-out-of-ground-effect,  low-level 
flight,  contour  flying,  nap-of-the-earth  flight,  and 
simulated  firing  missions  to  test  detectability. 
Test  results  supported  the  conclusion,  "that  the  generic 
AAH  performed  as  well  as,  or  better  than,  the  baseline 
AH- IS  and  was  judged  suitable  to  continuation  to  the 
next  phase  in  the  acquisition  cycle." 


10.4  OT&E  DURING  THE  FULL-SCALE  DEVELOPMENT  PHASE 

(Milestone  II  to  Milestone  III) 

The  OT&E  during  the  Full-Scale  Development  phase  is  conducted 
on  production  representative  systems.  These  operational  tests  estimate 
the  actual  operational  effectiveness  and  suitability,  and  ensure  that 
the  system  meets  operational  thresholds.  Just  prior  to  the  Full-Rate 
Production/Deployment,  Milestone  III  (MS- III)  decision,  a  dedicated  test 
and  evaluation  is  conducted  on  equipment  that  has  been  formally  certified 
by  the  Program  Manager  as  being  ready  for  the  "final  OT&E"  before  the 
full-rate  production/deployment  decision.  This  dedicated  OT&E  is  conducted 
in  as  operationally  realistic  test  environment  as  possible. 

10,4.1  OT&E  Objectives 

The  OT&E  tests  conducted  prior  to  the  full -rate  production 
decision  are  characterized  by  testing  performed  using  organizational 
units  in  controlled  field  exercises  to  examine  the  organization  and 
doctrine,  integrated  logistics  support,  threat,  communications,  command 
and  control,  and  tactics  associated  with  the  operational  employment  of 
the  unit,  under  continuous  tactical  operations.  This  OT&E  is  conducted 
to  support  the  Full-Rate  Production  Review,  MS- III,  and  includes  estimates 
which: 

(1)  Assess  operational  suitability  and  effectiveness; 

(2)  Assess  the  vulnerability/lethality  of  the  system; 

(3)  Assess  the  systems  reliability,  maintainability,  and  plans 
for  integrated  logistics  support; 

(4)  Evaluate  manpower,  personnel,  training  and  safety  requirements. 
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(5)  Validate  organizational  and  employment  concepts; 

(6)  Determine  training  and  logistics  requirements  deficiencies; 
and 

(7)  Assess  the  system's  readiness  to  enter  the  Full-Rate  Production/ 
Deployment  Phase. 

10.4.2  OT-II  of  the  Advanced  Attack  Helicopter 

The  AAH  OT-II  was  a  comparative,  three-phase  test 
conducted  at  Fort  Hunter-Liggett,  California.  Typical 
organizations  and  equipment  were  used  during  the 
test  and  operational  units  provided  personnel  and 
resources  for  both  the  AH-64  and  baseline  AH1S  aircraft 
sections.  The  AH1S  and  AH-64  aircraft  were  flown 
in  the  same  operational  and  threat  environment. 

The  three  phases  of  the  test  included  a  training 
phase,  non-live  fire  phase,  and  live  fire  phase. 
Appropriate  exploratory  trials  preceded  each  phase. 
Force-on-force  and  many-on-many  engagements,  with 
real  time  casualty  assessments,  were  conducted  during 
the  non-live  fire  phase.  The  live-fire  phase  included 
the  firing  of  all  of  the  AAH  weapons.  The  purpose 
of  the  OT-II  was  to  assess  the  military  effectiveness 
of  the  AH-64  against  the  baseline  aircraft.  The 
AH-64  was  also  evaluated  in  terms  of  RAM,  and 
supportability  in  the  operational  environment.  The 
report  of  the  OT-II  stated:  "the  performance  of  the 
AH-64  was  adequate  for  combat,  superior  to  the  present 
attack  helicopters,  night  capable,  and  survivable. ” 

There  were  no  operational  issues  which  were  considered 
to  preclude  the  acquisition  and  development  of  the 
system. 


10.5  OTAE  DURING  THE  FULL-RATE  PRODUCTION/DEPLOYMENT  PHASE 

(Milestone  III  to  Milestone  IV) 

If  a  program  is  large  and  there  is  a  significant  time  between 
the  beginning  of  low-rate  initial  production  and  full-rate  production, 
there  may  be  a  need  to  conduct  a  Service-base  Program  Review  or  OT- III 
before  a  decision  to  proceed  into  full-rate  production  can  be  made. 

After  the  MS- III  decision,  the  emphasis  shifts  towards 
procurement  of  production  quantities,  repairing  hardware  deficiencies, 
managing  changes,  and  phasing  in  full  logistics  support.  During  initial 
deployment  of  the  system,  the  0T8E  agency  and/or  the  user  may  perform 
Follow-On  Test  8  Evaluation  (F0T8E)  to  refine  the  effectiveness  and 
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suitability  estimates  made  during  earlier  OT&E. 

The  FOT&E  is  performed  on  production  articles,  by  an  operational 
command;  it  is  normally  funded  with  operational  and  maintenance  funds. 
The  FOT&E  conducted  during  this  phase  may  also  be  used  to: 

(1)  Ensure  that  the  production  system  performs  as  well  as  reported 
at  the  MS- III  review; 

(2)  Demonstrate  expected  performance  and  reliability  improvements; 
and 

(3)  Assure  that  the  correction  of  deficiencies  identified  during 
earlier  testing  have  been  completed. 

10.5.1  Objectives  of  Production  OT&E 

The  primary  objectives  of  OT&E  conducted  after  the  full -rate 
production  decision  are  to  refine  the  estimates  of  operational 
effectiveness  and  suitability,  to  identify  remaining  operational 
deficiencies,  to  clear  any  conditional  deficiencies  noted  at  the  Full-Rate 
Production  Review,  to  evaluate  systems  changes,  or  to  evaluate  the  system 
against  changing  operational  needs.  The  test  is  also  conducted  for  block 
revisions  to  a  system's  software  to  verify  sustained  software  improvements. 

10.6  OT&E  DURING  THE  LOGISTICS  READINESS  AND  SUPPORT  PHASE 
(Milestone  IV  to  Milestone  V) 

Testing  conducted  after  the  Milestone  IV  decision  will  encompass 
a  review  1  to  2  years  after  initial  operational  deployment  of  the  system. 
Operational  testing  is  conducted  to  assure  that  operational  readiness 
and  support  objectives  are  being  achieved  and  maintained  during  the  first 
several  years  of  the  system's  life.  The  testing  is  also  conducted  on 
production  equipment  that  includes  all  modifications,  product  improvements 
and  block  upgrades  to  determine  operational  readiness  and  supportability. 

10.6.1  FOT&E  After  the  Full-Rate  Production  Decision 

OT&E  can  also  be  the  follow-on  operational  test  and  evaluation 
that  is  conducted  on  systems  that  go  into  Full-Rate  Production  and  have 
not  been  through  an  independent  (Initial)  OT&E.  The  objectives  of  OT&E 
in  this  case,  are  to  validate  the  operational  effectiveness  and  suitability 
of  the  production  system  or  to  perform  an  operational  assessment  of  the 
system  in  new  environments,  in  different  platform  applications,  in  new 
tactical  applications,  or  against  new  threats. 

10.6.2  0T&E  Objectives 

The  testing  objectives  to  support  the  Logistics  Readiness  and 
Support  Phase  Review  are: 
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(1)  Assess  the  logistics  readiness  and  sustainability; 

(2)  Evaluate  the  weapon  support  objectives; 

(3)  Assess  the  implementation  of  integrated  logistics  support  plans; 

(4)  Evaluate  the  capability  of  the  logistics  support  activities; 

(5)  Determine  the  disposition  of  displaced  equipment;  and 

(6)  Evaluate  the  affordability  and  life-cycle  of  the  system. 

10.7  OT&E  SUPPORT  OF  OPERATION  SUPPORT 
(Milestone  IV  to  Milestone  V) 

After  the  Milestone  IV  review,  operational  testing  consists 
of  additional  FOT&E  as  required  to  refine  the  effectiveness  and  suitability 
estimates  and  to  confirm  that  deficiencies  noted  during  previous  tests 
have  been  corrected.  The  Milestone  V  review  is  conducted  5  to  10  years 
after  initial  deployment  of  the  system  to  determine  if  major  upgrades 
are  necessary,  or  if  existing  deficiencies  warrant  development  of  a 
replacement  system. 

OT&E  conducted  to  support  the  Milestone  V  review  is  conducted 
on  selected  product  improvements,  systems  modifications,  against  new 
threats,  or  whenever  deemed  necessary  to  determine  the  systems 
effectiveness  and  suitability  in  relation  to  its  tactical  employment 
environment.  The  objectives  of  the  FOT&E  during  the  Operational  and 
Support  Phase  are  to  determine: 

(1)  If  the  system  can  continue  to  meet  its  original  or  evolved 
mission  requirement; 

(2)  The  need  for  modifications  or  upgrades  to  ensure  that  mission 
requirements  will  be  met; 

(3)  If  the  system  is  capable  to  meet  changes  in  threat  that  require 
increased  capability  or  utility; 

(4)  If  changes  in  technology  have  occurred  that  present  an 

opportunity  for  a  significant  breakthrough  in  systems  worth;  and 

(5)  The  disposition  of  equipment  that  is  being  replaced. 

10.8  SUMMARY 

Operational  test  and  evaluation  is  that  T&E  (Operational 
Assessments  or  IOT&E)  or  FOT&E  conducted  to  estimate  a  system's  operational 
effectiveness  and  suitability,  identify  needed  modifications,  provide 
information  on  tactics,  doctrine,  organizations  and  personnel  requirements, 
and  evaluate  the  systems  logistic  supportabil ity.  The  acquisition  program 
should  be  structured  to  allow  operational  assessment  or  evaluation  to 
begin  early  in  the  development  cycle  and  to  continue  throughout  the 
system's  life  cycle. 


MODULE  IV 


SPECIALIZED  TESTING 


The  nature  of  a  weapon  system  sometimes  requires  the  use  of  a 
specially-tailored  test  and  evaluation  program.  In  some  cases,  hazardous 
testing  must  be  performed.  In  other  cases,  testing  must  be  conducted 
by  specialized  organizations  or  at  special  times  in  the  development  life 
cycle  . 

This  module  addresses  the  testing  of  special  weapons  (such  as 
chemical,  laser,  and  space  systems);  embedded  computer  systems;  electronic 
warfare  and  command  and  control  systems;  logistics  infrastructure  test 
&  evaluation,  and  production  related  testing  activities. 


CHAPTER  11 


TESTING  THE  SPECIAL  CASES 


11.1  INTRODUCTION 

This  chapter  covers  the  special  factors  and  alternative  test 

strategies  that  the  tester  must  consider  in  testing  dangerous  or  lethal 
weapons,  systems  that  involve  one-of-a-kind  or  limited  production,  and 
systems  with  high  cost  and/or  special  security  considerations.  Examples 
include  chemical  and  laser  weapons,  ships,  space  weapons,  missile  systems, 
and  electronic  warfare  (EW),  command  and  control  (C2),  and  Intelligence 

systems. 

11.2  TESTING  OF  HAZARDOUS  WEAPONS 

The  tester  of  dangerous  or  lethal  systems,  such  as  chemical 

and  laser  weapons,  must  consider  various  safety,  health,  and  medical 

factors  in  developing  his  test  plans,  such  as: 

(1)  Provision  of  medical  facilities  for  pre-  and  post-test 
checkups  and  emergency  treatment; 

(2)  Need  for  protective  gear  for  participating/observer 
personnel ; 

(3)  Approval  of  the  test  plan  by  Surgeon  General; 

(4)  Restrictions  in  selection  of  test  participants  (e.g., 

medical  criteria  or  use  of  only  volunteer  troops);  and 

(5)  Restricted  test  locations. 

(6)  Environmental  Impact  Statements 

Additionally  he  must  allow  for  additional  planning  time,  test  funds 
and  test  resources  to  accommodate  such  factors. 

11.2.1  Chemical  Weapons  Testing 

The  testing  of  chemical  weapons  poses  unique  problems  because 

the  tester  cannot  perform  actual  open  air  field  testing  with  real  nerve 
agents  or  other  toxic  chemicals.  Since  the  United  States  signed  and 

ratified  the  Geneva  Protocol  of  1925,  U.S.  policy  has  been  that  the  United 
States  will  never  be  the  first  to  use  lethal  chemical  weapons;  it  may, 

however,  retaliate  with  chemical  weapons  if  so  attacked.  In  addition 
to  the  health  and  safety  factors  discussed  in  the  last  paragraph,  the 
test  issues  that  the  chemical  weapons  tester  must  address  include: 

(1)  All  possible  chemical  reactions  due  to  variations  such 
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as  moisture,  temperature,  pressure,  and  contamination; 

(2)  Physical  behavior  of  the  chemical,  i.e.,  droplet  size, 
dispersion  density,  and  ground  contamination  pattern, 
when  used  operational ly; 

(3)  Toxicity  of  the  chemical,  i.e.,  lethality  and  duration 
of  contamination,  when  used  operational ly;  and 

(4)  Safety  of  the  chemical  weapon  during  storage,  handling, 
and  delivery. 

(5)  Decontamination  Process 

Addressing  all  of  these  issues  requires  a  combination  of  laboratory 
toxic  chamber  tests  and  open  air  field  testing.  The  latter  must  be 
performed  using  "simulants",  which  are  substances  that  replicate  the 
physical  and  chemical  properties  of  the  agent,  but  with  no  toxicity. 

The  development  and  use  of  simulants  for  testing  is  an  area  which 
wi 7 1  require  increased  attention  as  more  chemical  weapons  are  developed. 
Chemical  agents  can  demonstrate  a  wide  variety  of  effects  depending  on 
such  factors  as  moisture,  temperature,  and  contamination.  Consequently, 
the  simulants  must  be  able  to  replicate  all  possible  agent  reactions; 
it  is  likely  that  several  simulants  would  have  to  be  used  in  a  test  to 
produce  all  predicted  agent  behaviors.  In  developing  and  selecting 
simulants,  the  tester  must  thoroughly  understand  all  of  the  chemical 
and  physical  properties  and  possible  reactions  of  the  agent  as  possible. 

Studies  of  the  anticipated  reactions  can  be  performed  in  toxic  chamber 
tests  using  the  real  agent.  Here,  such  factors  as  changes  in  moisture, 
temperature,  pressure,  and  levels  of  impurity  can  be  controlled  to  assess 
the  agent's  behavior.  But,  the  tester  must  think  through  all  possible 
environmental  conditions  in  which  the  weapon  could  operate,  so  that  all 
cases  can  be  tested  in  the  laboratory  chamber  with  the  real  agent.  For 
example,  during  development  testing  of  the  BIGEYE  chemical  weapon,  it 
was  found  that  higher  than  expected  temperatures  due  to  aerodynamic  heating 
caused  pressure  buildup  in  the  bomb  body  that  resulted  in  the  bomb 
exploding.  This  caused  the  operational  concept  for  the  BIGEYE  to  be 
changed  from  onboard  mixing  of  the  two  chemicals  to  mixing  after  release 
of  the  bomb. 

Tests  to  confirm  toxicity  must  be  conducted  in  the  actual  environment 
using  simulants.  Since  the  agent's  toxicity  is  dependent  on  such  factors 
as  droplet  size,  dispersion  density,  ground  contamination  pattern,  and 
degradation  rate,  a  simulant  that  behaves  like  the  agent  must  be  used 
in  actual  field  testing.  Agent  toxicity  is  determined  in  the  lab. 
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The  Services  publish  a  variety  of  technical  documents  on  specific 
chemical  test  procedures.  Documents  such  as  the  U.S.  Army  Test  and 
Evaluation  Command  (TECOM)  Pamphlet  310-4,  which  is  a  bibliography  that 
includes  numerous  reports  on  chemical  testing  issues  and  procedures, 
can  be  consulted  for  specific  documentation  on  chemical  testing. 

11.2.2  Laser  Weapons  Testing 

Many  new  weapon  systems  are  being  designed  with  embedded  laser 
rangefinders  and  laser  designators.  Because  of  the  danger  to  the  human 
eye  posed  by  lasers,  the  tester  must  adhere  to  special  safety  requirements 
and  utilize  special  locations  during  T&E.  For  instance,  the  only  Army 
installation  in  the  continental  United  States  permitting  free  play  airborne 
laser  testing  is  Ft.  Hunter-Liggett;  during  tests  involving  lasers,  the 
airspace  must  be  restricted  and  guards  must  be  posted  to  prevent  anyone 
from  accidently  venturing  into  the  area.  A  potential  solution  to  the 
safety  issue  is  to  develop  and  use  an  "eye-safe"  laser  for  testing. 
The  tester  must  ensure  that  eye-safe  lasers  produce  the  same  laser  energy 
as  the  real  laser  system. 

Another  concern  of  the  laser  energy  weapons  tester  is  the 
accurate  determination  of  laser  energy  level  and  location  on  the  target. 
Measurements  of  the  laser  energy  on  the  target  are  usually  conducted 
in  the  laboratory  as  part  of  DT.  In  the  field,  video  cameras  are  often 
used  to  verify  that  the  laser  designator  did  indeed  illuminate  the  target. 
Such  determinations  are  important  when  the  tester  is  trying  to  attribute 
weapon  performance  to  behavior  of  the  laser,  behavior  of  the  guidance 
system,  or  some  other  factor. 

TECOM  Pamphlet  310-4,  a  bibliography  of  Army  test  procedures, 
lists  several  documents  which  cover  the  special  issues  associated  with 
laser  testing. 

11.3  SPACE  SYSTEM  TESTING 

From  a  historical  perspective,  space  system  acquisition  has 
posed  several  unique  problems  to  the  test  process  (especially  the 
operational  test  process)  which  generally  fall  into  four  categories; 
limited  quantities/high  cost;  "block  upgrade"  approach  to  acquisition; 
operating  environment  (peacetime  and  wartime);  and  test  environment. 

(1)  Limited  quantities/hiqh  cost  -  Space  systems  have 
traditionally  involved  the  acquisition  of  a  relatively  few  (historically 
less  than  20)  systems  at  extremely  "high  per  unit  costs"  (in  comparison 
with  more  traditional  military  systems).  The  high  per  unit  costs  are 
driven  by  a  combination  of  high  transportation  costs  (launch  to  orbit); 
high  life-cycle  reliability  requirements  and  associated  costs  because 
of  the  lack  of  an  "on-orbit"  maintenance  capability;  and  the  high  costs 
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associated  with  the  "leading  edge"  technologies  that  tend  to  be  a  part 
of  spacecraft  design.  From  a  test  perspective,  this  tends  to  drive  space 
system  acquisition  strategy  into  the  "non-standard"  approach  addressed 
in  paragraph  11.4  below.  The  problem  is  compounded  by  the  "block  upgrade" 
approach  to  acquisition  addressed  in  the  next  paragraph. 

(2)  Block  upgrade  approach  to  acquisition  -  Due  to  the  "limited 
buy"  and  "high  per  unit  cost*  nature  of  spacecraft  acquisition,  these 
systems  tend  to  be  procured  using  a  "block  upgrade"  acquisition  strategy. 
Under  this  concept,  "the  decision  to  deploy"  is  very  often  made  at  the 
front  end  of  the  acquisition  cycle  and  the  first  prototype  to  be  placed 
in  orbit  becomes  the  first  operational  asset.  As  early  and  follow-on 
systems  undergo  both  ground  and  on-orbit  testing  (either  DT&E  or  OT&E), 
discrepancies  are  corrected  by  "block  changes"  to  the  next  system  in 
the  pipeline.  This  approach  to  acquisition  can  perturb  the  test  process 
in  that  the  tester  may  not  have  any  formal  milestone  decisions  to  test 
towards.  His  focus  must  change  toward  being  able  to  influence  the  design 
of  (and  block  changes  to)  systems  further  downstream  in  the  pipeline. 
Additionally,  the  fact  that  the  first  "on-orbit"  asset  usually  becomes 
the  first  operational  asset  creates  pressure  from  the  operational  community 
to  expedite  (and  sometimes  limit)  testing  so  that  a  limited  operational 
capability  can  be  declared  and  the  system  can  begin  fulfilling  mission 
requirements.  Once  the  asset  "goes  operational,"  any  use  of  it  for  testing 
will  have  to  compete  with  operational  mission  needs  -  a  situation  where 
the  tester  may  find  himself  in  a  positon  of  relatively  low  priority. 
Recognition  of  these  realities  and  careful  "early-on"  test  planning  can 
overcome  many  of  these  problems,  but  the  tester  needs  to  be  involved 
and  ready  much  earlier  in  the  cycle  than  with  traditional  systems. 

(3)  Operating  environment  (peacetime  and  wartime)  -  Most 
currently  deployed  space  systems  and  near-term  future  space  systems  operate 
in  the  military  support  arena  such  as  tactical  warning/attack  assessment, 
communications,  navigation,  weather,  and  intelligence,  and  their  day-to-day 
peacetime  operating  environment  is  not  much  different  from  the  wartime 
operating  environment  except  for  activity  level  (i.e.,  message  throughput, 
more  objects  to  track/see  etc.)  Historically  space  has  also  been  a 
relatively  benign  battlefield  environment  because  of  technology  limitations 
in  the  capability  of  potential  adversaries  to  reach  into  space  with 
weapons.  This  combination  of  support  type  missions  and  a  battlefield 
environment  that  is  not  much  different  from  the  peacetime  environment 
has  played  a  definite  role  in  allowing  systems  to  reach  limited  operational 
capability  without  as  much  dedicated  prototype  system  level  testing  as 
one  might  see  on  other  type  systems.  However,  this  situation  is  likely 
to  change  with  the  advent  of  systems  such  as  the  Strategic  Defense 
Initiative  (SOI)  where  we  will  then  have  large  numbers  of  actual  weapons 
systems  sitting  passively  on  alert  in  space  and  where  day-to-day  peacetime 
operations  will  not  be  as  much  of  a  mirror  image  of  the  anticipated 
battlefield  environment.  Likewise,  the  elevation  of  the  battlefield 
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into  space  and  the  advancing  technologies  which  allow  potential  adversaries 
to  reach  into  space  may  also  change  the  thrust  of  how  space  systems  need 
to  be  tested  in  space.  One  might  expect  to  see  more  of  a  need  for 
dedicated  on-orbit  testing  on  some  type  of  a  space  range  where  the 
battlefield  environment  can  be  replicated  -  a  situation  similar  to  the 
dedicated  testing  done  today  on  test  ranges  with  Army,  Navy,  and  Air 
Force  weapons. 

(4)  Test  Environment  -  The  location  of  space  assets  in  "remote" 
orbits  also  compounds  the  test  problem.  Space  systems  simply  do  not 
have  the  ready  access  (as  with  ground  or  aircraft  systems)  to  correct 
deficiencies  identified  during  testing.  This  situation  has  driven  the 
main  thrust  of  testing  into  the  "pre-launch"  ground  simulation  environment 
where  discrepancies  can  be  corrected  before  the  system  becomes 
inaccessible.  However,  as  indicated  in  the  previous  paragraphs,  as  space 
system  missions  change  from  a  war  support  focus  to  a  war  fighting  focus, 
and,  as  the  number  of  systems  required  to  do  the  mission  increases  out 
of  the  "high  reliability/limited  number"  mode  into  a  more  traditional 
"fairly  large  number  buy"  mode,  then  one  would  expect  future  space  system 
testing  to  become  more  like  that  associated  with  current  ground,  sea, 
and  air  systems.  From  a  test  perspective,  this  could  also  create  unique 
"test  technology"  requirements  in  that  with  these  systems  we  will  have 
to  bring  the  test  range  to  the  operating  system  as  opposed  to  bringing 
the  system  to  the  range.  Lastly,  because  the  space  environment  tends 
to  be  "visible  to  the  world"  (others  can  observe  our  tests  as  readily 
as  we  can),  unique  test  operations  security  methodologies  may  be  required 
to  allow  us  to  achieve  test  realism  without  giving  away  system 
vulnerabilities. 

In  summary,  current  and  near  term  future  space  systems  have 
unique  test  methodologies.  However,  looking  toward  the  future,  where 
space  operations  might  entail  development/deployment  of  large  numbers 
of  weapon  platforms  on  orbit  with  lower  design  life  reliability  (because 
of  cost)  and  where  day-to-day  peactime  operations  do  not  mirror  the  wartime 
environment,  space  system  testing  requirements  may  begin  to  more  closely 
parallel  those  of  more  traditional  weapon  systems. 

11.4  TESTING  WITH  LIMITATIONS 

Certain  types  of  systems  cannot  be  tested  using  relatively 
standard  T&E  approaches  for  reasons  such  as  a  non-standard  acquisition 
strategy,  resource  limitations,  or  cost,  safety,  or  security  constraints. 
The  TEMP  must  contain  a  statement  that  identifies  "those  factors  that 
will  preclude  a  full  and  completely  realistic  operational  test...  (IOTSE 
and  FOT&E),"  such  as  inability  to  realistically  portray  the  entire  threat, 
limited  resources  or  locations,  safety,  and  system  maturity.  The  impact 
of  these  limitations  on  the  test's  critical  operational  issues  must  also 
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be  addressed  in  the  ’P. 


Non-standard  acquisition  strategies  are  often  used  for  one-of-a- 
kind  or  limited  production  systems.  Examples  of  these  include  space 
systems,  missiles,  ships,  electronic  warfare  (EW),  C3,  and  intelligence 
systems.  For  one-of-a-kind  systems,  the  production  decision  is  often 
made  prior  to  system  design,  hence  testing  does  not  support  the  traditional 
decision  process.  In  limited  production  systems,  there  are  often  no 
prototypes  available  for  test,  so  the  tester  must  develop  innovative 
test  strategies.  DoDD  5000.3  states: 

"for  these  programs,  the  principle  of  DT&E  of 
components,  subsystems,  and  prototype  or  first 
production  models  of  the  system  shall  be  applied. 

For  these  special  systems,  the  Component  [Operational 
Test  Agency]  OTA  shall  monitor  and  participate  In 
relevant  laboratory  and  controlled  testing,  and  use 
these  test  results,  as  appropriate,  to  provide  an 
assessment  of  system  effectiveness  and  suitability. 
Compatibility  and  interoperability  with  existing 
or  planned  equipment  shall  be  tested  during  DT&E 
and  OT&E.  After  production  of  the  system,  the 
Component  OTA  (or  user,  with  the  concurrence  of  the 
OTA)  shall  conduct  a  rigorous  operational  test  and 
provide  an  evaluation,  as  appropriate,  to  provide 
an  assessment  of  system  effectiveness  and  suitability 
in  the  same  manner  as  for  more  typical  systems." 

11.5  OPERATIONS  SECURITY  AND  T*E 

Operations  security  (OPSEC)  issues  must  be  considered  in  all 
test  planning.  DoDD  5000.3  requires  the  protection  of  "sensitive  design 
information  and  test  data"  throughout  the  acquisition  cycle  by, 

(1)  Protection  of  sensitive  technology...; 

(2)  Elimination  of  nonsecure  transmittal  data  on  and  from 
test  ranges;  and 

(3)  Providing  secure  communications  linking  DoD  agencies  to 
each  other  and  to  their  contractors." 

Such  protection  is  obviously  costly  and  will  require  additional  planning 
time,  test  resources,  and  test  constraints.  The  test  planner  must 
determine  all  possible  ways  in  which  the  system  could  be  susceptible 
to  hostile  exploitation  during  testing.  For  example,  announcement  of 
test  schedule  and  location  could  allow  monitoring  by  unauthorized  persons. 
Knowledge  of  the  locations  of  systems  and  instrumentation  or  test  concepts 
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could  reveal  classified  system  capabilities  or  military  concepts. 
Compilations  of  unclassified  data  could,  as  a  whole,  reveal  classified 
Information,  as  could  surveillance  (electronic  or  photographic)  of  test 
activities  or  intercept  of  unencrypted  transmissions.  The  test  and 
evaluation  regulations  of  each  Service  require  an  operational  security 

plan  for  a  test.  APR  55-43  provides  a  detailed  list  of  questions  the 
test  planner  can  use  to  identify  the  potential  threat  of  exploitation. 

11.6  SUMMARY 

All  weapon  systems  tests  are  limited  to  one  degree  or  another, 
but  certain  systems  face  major  limitations  that  could  preclude  a  full 

and  realistic  test.  The  test  planners  of  these  special  systems  must 

allow  additional  planning  time,  budget  for  extra  test  resources,  and 

devise  alternative  test  strategies  to  work  around  testing  limitations 
caused  by  such  factors  as  security  restrictions,  resource  availability, 
environmental  safety  factors,  and  non-standard  acquisition  strategies. 
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CHAPTER  12 

EMBEDDED  COMPUTER  SYSTEMS  TESTING 


12. 1  INTRODUCTION 

Software  components  present  a  major  development  risk  for  military 
computer  systems.  They  escalate  the  cost  and  reduce  the  reliability 
of  military  systems.  Embedded  computer  systems  whose  major  function 
is  not  data  processing  are  physically  incorporated  into  larger  systems 
whose  major  function  is  not  data  processing.  The  outputs  of  the  systems 
are  normally  information,  control  signals  or  computer  data  required  by 
the  host  system  to  complete  its  mission.  Although  hardware  and  software 
contribute  in  equal  measure  to  successful  implementation  of  embedded 
computer  system  functions,  there  have  been  relative  imbalances  in  their 
treatment  during  system  development. 

The  development  of  embedded  systems  involves  a  series  of 
activities  in  which  there  are  frequent  opportunities  for  errors.  Errors 
may  occur  at  the  inception  of  the  process  when  the  requirements  of  the 
system  may  be  erroneously  specified  or  later  in  development  cycle  when 
system  specifications  are  implemented.  This  chapter  will  address  the 
use  of  testing  to  control  the  development  risk  of  embedded  computer 
systems,  particularly  as  it  pertains  to  the  software  development  process. 


12.2  MISSION  CRITICAL  COMPUTER  RESOURCES 

The  term  Mission  Critical  Computer  Resources  (MCCR)  is  defined 
as  automated  data  processing  equipment,  software  or  services  where  the 
function,  operation  or  use  of  the  equipment  software  or  services: 

(1)  involves  intelligence  activities; 

(2)  involves  cryptologic  activities  related  to  national 
security; 

(3)  involves  command  and  control  of  military  forces; 

(4)  involves  equipment  that  is  an  integral  part  of  a  weapons 
system;  or 

(5)  is  critical  to  the  direct  fulfillment  of  military  or 
intelligence  missions. 

Acquisition  of  MCCR  is  defined  by  DoD  Directive  5000.29, 
Management  of  Computer  Resources  in  Major  Defense  Systems,  which  requires 
the  validation  of  computer  resource  requirements  prior  to  Milestone  II, 
to  ensure  conformance  with  stated  operational  requirements.  After 
Milestone  II  computer  resources  life  cycle  planning  must  continue  to 
ensure  adequate  personnel,  system  integration,  quality  and  integrity. 
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12.3  PURPOSE  OF  SOFTWARE  TEST  AND  EVALUATION 

A  major  problem  in  software  development  is  a  lack  of  well  defined 
requirements.  If  requirements  are  not  well  defined,  errors  can  multiply 
throughout  the  development  process.  As  is  illustrated  in  Figure  12-1, 
errors  may  occur  at  the  very  inception  of  the  process  during  requirements 
definition,  when  objectives  may  be  erroneously  or  imperfectly  specified, 
during  the  later  design  and  development  stages,  when  these  objectives 
are  implemented;  as  well  as  during  software  maintenance  and  operational 
phases  when  software  changes  are  needed  to  eliminate  errors  or  enhance 
performance. 

Current  estimates  of  increased  software  costs  arising  from 
incomplete  testing  help  to  illustrate  the  dimension  of  software  life-cycle 
costs.  Averaged  over  the  operational  life  cycle  of  a  computer  system, 
development  costs  encompass  approximately  30  percent  of  total  system 
costs.  The  remaining  70  percent  of  life-cycle  costs  are  associated  with 
maintenance  which  includes  system  enhancements  and  error  correction. 
More  complete  testing  during  earlier  development  phases  may  have  detected 
these  errors. 

The  relative  costs  of  error  correction  increases  as  a  function 
of  time  from  the  start  of  the  development  process.  Relative  costs  of 
error  correction  rises  dramatically  between  requirements  and  design  phases 
and  then  even  more  dramatically  during  code  implementation  (a 
representative  cost  function  is  shown  in  Figure  12.2). 

Previous  research  in  the  area  of  software  T&E  reveals  that 
half  of  all  maintenance  costs  are  incurred  in  the  correction  of  previously 
undetected  errors.  Approximately  one  half  of  the  operational  life  cycle 
costs  can  be  traced  directly  to  inadequate  or  incomplete  testing 
activities.  In  addition  to  cost  increases,  operational  implications 
of  software  errors  in  weapon  systems  can  result  in  mission  critical 
software  failures  which  may  impact  mission  success  and  personnel  safety. 

A  more  systematic  and  rigorous  approach  to  software  testing 
is  required.  To  be  effective,  this  approach  must  be  applied  to  all  phases 
of  the  development  process  in  a  planned  and  coordinated  manner,  beginning 
at  the  earliest  design  stages  and  proceeding  through  operational  testing 
of  the  integrated  system.  Early  detailed  software  test  and  evaluation 
(T&E)  planning  is  critical  to  the  successful  development  of  a  computer 
system. 

12.4  SOFTWARE  DEVELOPMENT  PROCESS 

A  success  oriented  software  development  methodology  is 
characterized  by  a  phased  approach  starting  with  a  detailed  requirements 
analysis.  Key  elements  of  this  methodology  include  a  top  down  software 
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NEED  I  THE  ERROR  AVALANCHE 
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Figure  12-1.  The  Error  Avalanche 
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design,  early  design  and  configuration  management  controls,  frequent 
milestone  reviews  and  the  continuous  use  of  software  documentation  as 
a  means  of  controlling  and  monitoring  the  development  team  and  tracking 
the  interdependence  of  the  software  modules.  DoD  Standard  2167A 
establishes  requirements  to  be  applied  during  development  and  acquisition 
of  software  in  Mission  Critical  Computer  Resources  Systems.  The  software 
development  cycle  in  DoD  STD  2167A  is  divided  into  six  consecutive  phases: 

(1)  Software  requirements  analysis, 

(2)  Preliminary  software  design, 

(3)  Detailed  software  design, 

(4)  Coding  and  unit  testing, 

(5)  Computer  Software  Component  (CSC)  Integration  and  Testing, 
and 

(6)  Computer  Software  Configuration  Item  (CSCI)  testing. 

Each  phase  is  concerned  with  a  specific  aspect  of  the  overall 
software  development  activity.  In  general,  the  phases  are  separated 
by  milestone  reviews.  These  reviews  occur  at  the  end  of  a  particular 
phase  and  provide  a  means  of  formally  monitoring  progress.  The  production 
of  software  documentation  goes  on  throughout  the  phases.  Figure  12.3 
illustrates  the  major  documentation  throughout  the  development  process. 
This  approach,  in  conjunction  with  unit  development  folders  (UDFs),  assist 
in  maintaining  a  clearly  definable  work  breakdown  structure  (WBS)  for 
successfully  managing  a  software  development  effort.  These  UDFs  contain 
all  relevant  material  related  to  each  identified  software  module. 

Software  engineering  technologies  used  to  produce  operational 
software  are  key  risk  factors  in  a  development  program.  The  TEMP  is 
should  determine  which  of  these  technologies  increase  risk  and  have  a 
life  cycle  impact.  A  principal  source  of  risk  is  the  support  software 
required  to  develop  operational  software.  In  terms  of  life  cycle  impact, 
a  common  source  of  operational  software  problems  are  associated  with 
the  difficulty  in  maintaining  and  supporting  the  software  once  deployed. 
Software  assessment  requires  an  analysis  of  the  life  cycle  impact  which 
varies  depending  on  the  technology  used  to  design  and  implement  the 
software.  One  approach  to  reducing  the  long-term  life  cycle  risks  is 
to  use  common  hardware  throughout  the  development  and  operation  of  the 

software.  These  life  cycle  characteristics  which  affect  operational 
capabilities  must  be  addressed  in  the  TEMP  and  tests  should  be  developed 
to  test  problems  that  these  characteristics  raise.  The  technology  used 

to  design  and  implement  the  software  may  significantly  affect  software 

supportabil ity  and  maintainability.  As  an  example.  High  Order  Languages 
(HOL)  that  have  yet  undemonstrated  application  are  a  frequent  source 
of  risk.  The  first  use  of  an  HOL  increases  the  risk  in  three  dimensions: 

(1)  A  "learning  curve"  effect  limits  the  productivity  of 

software  development  team  in  the  early  phases. 
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(2)  The  immaturity  of  the  HOL  compiler  increases  the  probability 
that  software  errors  may  be  introduced  during  the 
implementation. 

(3)  The  new  HOL  may  not  have  demonstrated  suitability  for 
the  given  application. 

The  TEMP  must  sufficiently  describe  the  acceptance  criteria 
for  the  written  specifications,  operational  suitability  and  effectiveness. 
The  specifications  must  define  the  required  software  characteristics 
in  order  to  set  goals  and  thresholds  for  mission  critical  functions. 
Additionally,  these  characteristics  should  be  evaluated  at  the  appropriate 
stage  of  system  development,  rather  than  at  some  arbitrarily  imposed 
milestone. 

12.5  TtE  IN  THE  SOFTWARE  LIFE  CYCLE 

Software  testing  is  defined  as  a  controlled  exercise  of  program 
code  to  expose  errors.  Software  test  planning  should  be  described  in 
the  TEMP  with  the  same  care  as  test  planning  for  other  system  components. 

12.5.1  TESTING  APPROACH 

While  software  is  normally  developed  in  a  top-down  approach, 

all  testing  is  normally  performed  from  the  bottom  up.  The  smallest 
controlled  software  modules,  the  units,  are  tested  individually.  They 
are  then  combined  or  integrated  and  tested  in  larger  aggregate  groups 
or  "builds".  When  this  process  is  complete,  the  software  system  is  tested 
in  its  entirety  during  the  Developmental  Test  and  Evaluation  phase  of 

the  test  program.  The  tested  and  approved  system  is  known  as  the  Initial 
Operational  Configuration. 

The  usual  software  test  program  first  verifies  the  adherence 

of  the  code  to  the  detailed  design  (unit  test),  then  to  the  top  level 

design  (integration),  and  finally  to  the  system  requirements  themselves 
(developmental  test).  The  operational  test  further  validates  the  adherence 
of  the  code  to  the  basic  system  requirements ,  as  well  as  validating  the 
requirements  themselves  against  the  real  world  environment.  Obviously, 
as  errors  are  found  in  the  latter  stages  of  the  test  program,  it  requires 
a  return  to  earlier  portions  of  the  development  program  to  provide 
corrections.  The  cost  impact  of  error  detection  and  correction  can  be 
diminished  using  the  bottom  up  testing  approach. 

Software  evaluation  will  be  included  in  the  assessment  of  the 
overall  system  suitability  and  effectiveness  during  critical  milestone 
reviews.  This  is  especially  important  for  the  test  and  evaluation  of 
software  embedded  within  mission  critical  computer  resources  belonging 
to  major  weapons  systems. 
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It  is  critical  that  adequate  software  T&E  information  be 
contained  in  documents  such  as  TEMPS  and  test  plans.  The  TEMP  must  define 
characteristics  of  critical  software  components  that  effectively  address 
goals  and  thresholds  for  mission  critical  functions.  The  measures  of 
effecti veness  (MOEs)  must  support  the  critical  software  issues.  The 
test  plan  should  specify  the  test  methodologies  which  will  be  applied. 
Test  methodologies  consist  of  two  components.  The  first  is  the  test 

strategy  which  guides  the  overal  testing  effort  and  the  second  is  the 
testing  technique  which  is  applied  within  the  framework  of  a  test  strategy. 

12.5.2  RAM/HUMAN  FACTORS  EVALUATION 

A  formal  assessment  of  reliability,  availability  and 

maintainability  (RAM)  must  be  planned  for  each  phase  of  development  and 
operational  testing.  Important  software  characteristics  include: 

(1)  Reliability:  This  characteristic  is  often  a  key  indicator 
of  software  suitability.  It  is  very  important  to  choose 
measurement  criteria  that  adequately  reflect  software 

reliability.  Interim  reliability  goals  must  be  established 
for  each  acquisition  phase  so  that  reliability  requirements 
can  be  assessed  during  the  entire  acquisition  process. 
Since  time  dependent,  reliability  goals  are  not  really 
meaningful  for  software,  goals  should  be  stated  in  terms 

of  observed  time  between  operational  mission  failure. 

These  measures  include,  but  are  not  limited  to: 

(a)  Mean-Time-Between-Operational-Mission  Failure  (MTBOMF) 

This  measure  can  be  used  as  an  indicator  of  system 
suitability  prior  to  deployment  and  as  a  control 

measure  to  track  corrective  actions,  after  the  start 
of  system  level  testing.  For  the  measure  to  be 
meaningful,  evaluation  criteria  must  be  established 
for  each  system  load  level. 

(b)  Mean-Time-Between-Loss-of-Function  (MTBLOF)  -  This 

Ts  a  measure  of  functional  reliability,  measuring 
the  time  between  occurrences  when  software  does  not 
perform  a  designed  function. 

(c)  Mean-Time-To-Restore  Function  (MTTRF)  -  This  measures 
time  between  loss  of  function  and  restoration  of 
that  function.  In  terms  of  software,  it  is  not  a 
corrective  action,  but  rather  a  restorative  function 
that  requires  maintenance  or  software  input  change. 
Corrective  action  may  be  required  to  prevent  recurrence 
of  the  loss  of  function. 

(d)  Percent  of  Functions  Verified  and  Validated  -  This 
measure  indicates  the  degree  software  and  system 
function  have  been  tested.  In  order  to  achieve  a 
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statistical  confidence  level  through  probability 
distribution,  the  true  operational  distribution  of 
the  inputs  must  be  known. 

(2)  Availability  and  Maintainability:  Hardware-oriented 

definitions  o?  availability  and  maintainability  are 
generally  not  applicable  to  software  system.  This  is 
due,  in  large  part,  to  the  fact  that  hardware  oriented 
logistics  parameters  do  not  apply  to  software  availability. 
Maintainability  of  software  incorporates  repair  and 

re-engineering.  Usually  maintenance  is  carried  out  at 
a  Post  Deployment  Software  Support  (PDSS)  facility  and 
factors  limiting  mean  time  to  repair  tend  to  revolve  around 
communications  and  the  labor-intensiveness  of  the 

maintenance  process.  To  determine  maintainability,  the 
following  software  characteristics  must  be  evaluated: 

(a)  Modularity  -  For  ease  of  understanding  and 
modification,  is  the  software  logically  partitioned 
into  parts  or  components  with  few  and  simple 
connections  between  parts? 

(b)  Descriptiveness  -  For  understandability,  does  the 
source  code  and  its  documentation  discuss  the  software 
objectives,  assumptions,  inputs,  processing,  outputs, 
components,  and  revision  status? 

(c)  Consistency  -  Are  standards  and  conventions  followed 
in  documentation,  input/output  processing,  error 
processing,  module  interfacing,  module/variable  naming, 
etc.? 

(d)  Simplicity  -  To  keep  the  code  simple,  are  such 
characteristics  as  large  numbers  of  operators, 
operands,  and  nested  control  structures,  dynamic 
storage  allocation  and  recursive/reentrant  coding 
avoided? 

(e)  Expandability  -  is  flexibility  provided  through 

parameterization  of  constants  and  basic  data  structure 
sizes? 

(f)  Instrumentation  -  Does  the  software  provide  for  the 
inclusion  of  testing  aids  either  through  embedded 
test  code  or  through  a  support  software  system? 

(3)  Human  Factors:  As  an  indicator  of  operational  suitability, 
human  factors  can  be  evaluated  early.  The  use  of 
simulators,  prototype  hardware,  and  operator  personnel 
can  give  reliable  indications  of  software  suitability. 
Early  determination  of  deficiencies  allows  correction 
through  redesign  of  the  software.  Later  detection  of 
unsuitable  human  factors  in  the  software  can  raise  the 
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cost  of  correction  considerably.  An  analysis  of  human 
factors  requires  an  evaluation  of  software  operator-machine 
interface  in  a  way  which  will  yield  a  consistent, 
quantifiable  analysis  of  various  aspects  including: 

(a)  Assurability  -  Is  sufficient  redundancy  and 

acknowledgement  of  user  input  required  to  ensure 

accuracy? 

(b)  Controllability  -  Can  the  user/operator  make 

adjustments  to  such  items  as  level  of  detail  of  help 
information  and  output  report,  or  stop  processing 
when  incorrect  inputs  have  been  applied? 

(c)  Workload  reasonability  -  Can  the  operator  at  each 

position  or  workstation  stay  abreast  of  the  software 
processing,  supplying  inputs  and  recording  results 
as  necessary? 

(d)  Descriptiveness  -  Does  the  software  provide  adequate 
information  for  operator  understanding  and  use? 

(e)  Consistency  -  Are  identical  operator-machine  interface 
conventions  used  at  each  operator  station? 

(f)  Simplicity  -  Are  user  options  useable  and  not 

excessive?  Are  error  diagnostics  straightforward? 
It  is  also  important  that  personnel  responsible  for 
evaluating  software  documention  and  module  source 
listings  have  similar  backgrounds  to  those  who  will 

maintain  the  software. 

Effective  test  methodologies  require  realistic  software  test 
environments  and  scenarios.  The  test  scenarios  must  be  appropriate  for 

the  test  objectives.  That  is,  are  the  test  results  interpretable  in 
terms  of  software  test  objectives.  The  test  scenarios  and  analysis  should 
actually  verify  and  validate  accomplishment  of  requirements.  The  test 
environments  must  be  chosen  on  a  careful  analysis  of  characteristics 

to  be  demonstrated  and  its  relationship  to  the  development,  operational 
and  support  environments.  In  addition,  environment  must  also  be 

representative  of  the  environment  in  which  the  software  will  actually 
be  maintained. 

12.6  SOFTWARE  QUALITY  ASSURANCE 

A  comprehensive  software  quality  program  plan  (SQPP)  provides 
an  effective  method  of  controlling  software  development  risk.  The  two 
major  characteristics  of  software  quality  are  suitability  and 
maintainability.  Below  this  there  are  sub-characteristics,  as  shown 
in  the  decision  tree  in  Figure  12-4.  There  are  both  proactive  and  reactive 
approaches  to  software  quality.  Both  approaches  need  to  be  applied  for 
any  MCCR  or  software  intensive  system.  The  proactive  approach  prevents 
error  propagation  through  the  following  means: 
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Figure  12-4.  Software  Quality  Characteristics 
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o  Methods 

oo  Structured  System  Analysis 

oo  Structured  Design 

oo  Structured  Programming 

o  Practices 

oo  Program  Library 

oo  Change  Control 

oo  Coding  Standards 

o  Tools 

oo  Software  Development  Facility 
oo  Higher  Order  Language 

oo  Diagnostic  Compiler 

oo  Code  Standards  Auditor 

The  reactive  approach  focuses  on  improving  software  quality 
once  code  is  being  or  has  been  developed.  This  approach  is  achieved 
through  the  following  methods,  practices  and  tools: 

o  Methods 

oo  Requirements  Modeling 

oo  Simulation 

oo  Interface  Analysis 

oo  Traceability  Analysis 

o  Practices 

oo  Design  Review  and  Walkthroughs 
oo  Code  Reading 

oo  Formal  Test 

o  Tools 

oo  Simulators 

oo  Dynamic  Debugging  Tools 

oo  Path  Analyzer 

The  principal  guide  for  software  quality  program  is  DoD-STD-2168, 
"Defense  System  Software  Quality".  This  specification  applies  to  software 
alone  and  to  software  as  part  of  other  systems.  The  software  quality 
program  requirements  under  include  addressing  the  following: 

1.  Work  tasking  and  authorization  procedures, 

2.  Configuration  management, 

3.  Testing  plan  and  procedures, 

4.  Corrective  actions, 

5.  Library  controls, 

6.  Critical  design  reviews. 
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7.  Software  documents, 

8.  Reviews  and  audits  and, 

9.  Tools,  techniques  and  methodologies. 

12.6.1  The  Software  Quality  Environment 

A  Software  Quality  Program  plan  must  be  capable  of  tracing 
requirements  through  the  development  process.  Traceability  tables  (Figure 
12-5)  are  used  to  document  and  trace  requirements  through  all  phases 
of  software  development.  These  tables  provide  a  tool  with  which  to 
identify  and  integrate  the  requirements  as  well  as  the  responsibilities, 
resources,  schedules  and  critical  issues  associated  with  an  embedded 
system.  This  analysis  should  also  include  a  summary  of  key  functions, 
interfaces  and  unique  characteristics  of  the  system  in  order  to  satisfy 
operational  objectives.  System  requirements  must  be  traced  through  the 
entire  T&E  process. 

The  advantages  of  performing  requirements  traceability  are: 

(1)  The  degree  to  which  the  design  follows  the  requirements 
can  be  assessed. 

(2)  Error  propagation  can  be  reduced. 

(3)  Any  requirement  can  be  traced  through  the  complete  system 

acquisition  process.  Figure  12.6  provides  an  illustration 
of  how  a  hierarchical  set  of  traceability  tables  or  matrices 
could  appear  in  the  system.  These  tables  would  serve 

as  an  index  into  documentation  providing  the  capability 
to  proceed  along  a  specific  thread  or  train  of  thought 
in  the  review  process. 

(4)  Misplaced  or  invalid  requirements  are  considered  anomalies 
and  can  be  detected  early  using  requirements  analysis 
and  matrices. 

To  perform  an  adequate  risk  assessment,  the  degree  to  which 
software  implements  critical  functions  must  be  evaluated.  The  primary 
concern  in  this  evaluation  is  to  ensure  that  the  software  has  been  given 
a  balanced  treatment  with  other  critical  system  components.  Those  software 
components  that  fulfill  a  requirement  for  a  Key  Function,  as  described 
in  the  TEMP,  should  be  identified  as  critical  software  components.  The 
risk  assessment  should  also  include  an  evaluation  of  the  support  software. 
As  part  of  this  effort,  a  software  assessment  correlation  matrix  should 
be  developed  which  maps  software  functional  specifications  to  technical 
risk  areas.  The  correlation  matrix  (or  equivalent  narrative)  is  the 
primary  source  of  information  about  how  capabilities  have  been  partitioned 
between  hardware  and  software.  These  partitions  will  be  important  in 
determining  required  characteristics,  in  defining  error/failure  categories, 
and  in  isolating  and  correcting  deficiencies  noted  during  testing. 
Therefore,  it  may  be  important  to  determine  that  proper  engineering  studies 
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Figure  12-5.  Traceability  and  Traceability  Anomalies 
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Figure  12-6.  Example  of  Traceability  Tables 
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have  led  to  the  establishment  of  these  partitions.  An  understanding 
of  the  sources  of  risk  in  each  of  the  software- implemented  functions 
identified  in  the  correlation  matrix  is  an  essential  part  of  the  overall 
risk  assessment. 

12.6.2  Independent  Verification  and  Validation 

Independent  verification  and  validation  ( IV&V )  is  a  risk  reducing 
technique  which  is  applied  to  major  software  development  efforts.  The 
primary  purpose  of  independent  verification  and  validation  is  to  ensure 
that  software  meets  requirements  and  is  reliable  and  maintainable.  IV&V 
is  only  effective  if  implemented  early  in  the  software  development 
schedule.  Requirements  analysis  and  risk  assessment  are  the  most  critical 
activities  performed  by  IV&V  organizations  their  effectiveness  is  limited 
if  brought  on  board  a  project  after  the  fact.  Often,  there  is  a  reluctance 
to  implement  IV&V  because  of  the  costs  involved,  but  early  implementation 
of  IV&V  will  result  in  lower  overall  costs  of  error  correction  and  software 
maintenance.  As  development  efforts  progress,  IV&V  involvement  typically 
decreases  due  more  to  the  expense  of  continued  involvement  rather  than 
a  lack  of  need.  For  an  IV&V  program  to  be  effective,  it  must  be  the 
responsibility  of  an  individual  or  organization  external  to  the  software 
development  program  manager. 

The  application  of  the  IV&V  process  to  software  development 
maximizes  the  maintainability  of  the  fielded  software  system,  while 
minimizing  the  cost  of  developing  and  fielding  it.  Maintenance  of  a 
software  system  falls  into  several  major  categories:  corrective 
maintenance,  modifying  software  to  correct  errors  in  operation;  adaptive 
maintenance,  modifying  the  software  to  meet  changing  requirements;  and 
perfective  maintenance,  modifying  the  software  to  incorporate  new  features 
or  improvements. 

The  IV&V  process  maximizes  the  reliability  of  the  software 
product,  which  eases  the  performance  of  and  minimizes  the  need  for 
corrective  maintenance.  It  also  attempts  to  maximize  the  flexibility 
of  the  software  product,  which  eases  the  performance  of  adaptive  and 
perfective  maintenance.  These  goals  are  achieved  primarily  by  determining 
at  each  step  of  the  software  development  process  that  the  software  product 
completely  and  correctly  meets  the  specific  requirements  determined  at 
the  previous  step  of  development.  This  step-by-step,  iterative  process 
continues  from  the  initial  definition  of  system  performance  requirements 
all  the  way  through  final  acceptance  testing. 

The  review  of  software  documentation  at  each  stage  of  development 
is  a  major  portion  of  the  verification  process.  The  current  documentation 
is  a  description  of  the  software  product  at  the  present  stage  of 
development  and  will  define  the  requirements  laid  on  the  software  product 
at  the  following  stage.  Careful  examination  and  analysis  of  the 
development  documentation  ensures  that  each  step  in  the  software  design 
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process  Is  consistent  with  the  previous  step.  Omissions,  Inconsistencies, 
or  design  errors  can  then  be  Identified  and  corrected  as  early  as  possible 
in  the  development  process. 

Continuing  participation  in  formal  and  informal  design  reviews 
by  the  I V&V  organization  maintains  the  communication  flow  between  software 
system  "customers"  and  developers,  assuring  that  the  software  design 
and  production  proceed  with  minimal  delays  and  misunderstandings.  Frequent 
informal  reviews,  design  and  code  walk-throughs  and  audits  ensure  that 
the  programming  standards,  software  engineering  standards,  software  quality 
assurance  and  configuration  management  procedures  designed  to  produce 
a  reliable,  maintainable  operational  software  system  are  indeed  followed 
throughout  the  process.  Continuous  monitoring  of  computer  hardware 
resource  allocation  throughout  the  software  development  process  also 
ensures  that  the  fielded  system  has  adequate  capacity  to  meet  operation 
and  maintainability  requirements. 

The  entire  testing  process,  from  the  planning  stage  through 
final  acceptance  test  is  also  approached  in  a  step-by-step  manner  by 
the  IV&V  process.  At  each  stage  of  development,  the  functional 
requirements  determine  test  criteria  as  well  as  design  criteria  for  the 
next  stage.  An  important  function  of  the  IV&V  process  is  to  ensure  that 
the  test  requirements  derive  directly  from  the  performance  requirements 
independent  of  design  implementation.  Monitoring  of,  participation  in, 
and  performance  of  the  various  testing  and  inspection  activities  by  the 
IV&V  contractor  ensure  that  the  developed  software  meets  requirements 
at  each  stage  of  development. 

Throughout  the  entire  software  development  process,  the  I VSV 
contractor  reviews  any  proposals  for  software  enhancement  or  change, 
proposed  changes  in  development  baselines  and  proposed  solutions  to  design 
or  implementation  problems  to  ensure  that  the  original  performance 
requirements  are  never  lost  sight  of.  An  important  facet  of  the  IV&V 
contractor's  role  is  to  act  as  the  objective  third  party  continuously 
maintaining  the  "audit  trail"  from  the  initial  performance  requirements 
to  the  final  operational  system. 

12.7  SUMMRY 

There  is  a  useful  body  of  software  testing  technologies  which 
can  be  applied  to  testing  of  embedded  systems.  As  a  technical  discipline, 
though,  software  testing  is  still  maturing.  Currently,  there  is  little 
to  guide  the  program  manager  in  choosing  one  testing  technique  over 
another.  It  is  apparent  that  systematic  test  and  evaluation  techniques 
are  far  superior  to  ad-hoc  testing  techniques.  Implementation  of  an 
effective  test  and  evaluation  plan  requires  a  set  of  strong  technical 
and  management  controls.  Given  the  increasing  number  of  embedded  computer 
systems  being  acquired,  there  will  be  an  increased  emphasis  on  tools 
and  techniques  for  test  and  evaluation. 
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CHAPTER  13 

EW/C2  TEST  AND  EVALUATION 


13. 1  INTRODUCTION 

Testing  of  electronic  warfare  (EW)  and  command  and  control 
(C2)  systems  poses  unique  problems  for  the  tester  because  of  the  difficulty 
in  measuring  their  performance  in  operational  terms.  Special  testing 
techniques  and  facilities  are  normally  required  in  EW  and  C2  testing. 
This  chapter  discusses  the  problems  associated  with  EW  and  C2  testing 
and  presents  methodologies  the  tester  can  consider  using  to  overcome 
the  problems. 

13.2  TESTING  EW  SYSTEMS 

13.2.1  Special  Consideration  When  Testing  EW  Systems 

The  purposes  of  EW  systems  are  to  increase  survivability,  degrade 

enemy  capability,  and  contribute  to  the  overall  success  of  the  combat 
mission.  Decision  makers  want  to  know  the  incremental  contribution  to 
total  force  effectiveness  made  by  a  new  EW  system,  which  is  a 
force-on-force  issue.  However,  the  contractual  specifications  for  EW 
systems  are  usually  stated  in  terms  of  engineering  parameters  such  as 
effective  radiated  power,  reduction  in  communications  intelligibility, 
and  jamming-to-signal  ratio;  these  measures  are  of  little  use  by  themselves 
in  assessing  contribution  to  mission  success.  The  decision  makers  require 
that  testing  be  conducted  under  realistic  operational  conditions,  but 
the  major  field  test  ranges,  such  as  the  shoreline  at  Eglin  AFB  or  the 
desert  at  Nellis  AFB,  cannot  provide  the  signal  density  or  realism  of 
threats  that  would  be  presented  by  a  Soviet  Combined  Arms  Army  in  the 
Fulda  Gap  in  Central  Europe.  In  field  testing,  the  tester  can  achieve 
one-on-one  or,  at  best,  few-on-few  testing  conditions.  To  do  this  he 
needs  a  methodology  that  will  permit  extrapolation  of  engineering 
measurements  and  one-on-one  test  events  to  create  more  operationally 
meaningful  measures  of  mission  success  in  a  force-on-force  context. 

13.2.2  Integrated  Test  Approach 

An  integrated  approach  to  EW  testing  using  a  combination  of 
large-scale  models,  computer  simulations,  hybrid  man-in-the-loop 
simulators,  and  field  test  ranges  is  a  solution  for  the  EW  tester.  No 
single  one  of  these  tools  by  itself  is  adequate  to  provide  a  comprehensive 
evaluation.  Simulation,  both  digital  and  hybrid,  can  provide  a  means 
for  efficient  test  execution.  Computer  models  can  be  used  to  simulate 
many  different  test  cases  to  aid  the  tester  in  assessing  the  critical 
test  issues  (i.e.,  sensitivity  analysis)  and  produce  a  comprehensive 
set  of  predicted  results.  As  digital  simulation  models  are  validated 
with  empirical  data  from  testing. 
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they  can  be  used  for  evaluation  of  the  system  under  test  in  a  more  dense 
and  complex  threat  environment  and  at  expected  wartime  levels.  In 
addition,  the  field  test  results  are  used  to  validate  the  model,  and 
the  model  is  also  used  to  validate  the  field  tests,  thus  lending  more 
credibility  to  both  results.  Hybrid  man-in-the-loop  simulators,  such 
as  the  Real-Time  Electromagnetic  Digitally  Controlled  Analyzer  and 
Processor  (REDCAP)  and  the  Air  Force  Electronic  Warfare  Evaluation 
Simulator  (AFEWES)  can  provide  a  capability  to  test  against  new  threats. 
Hybrid  simulators  are  also  cheaper  and  safer  than  field  testing.  The 
field  test  ranges  are  used  when  a  wider  range  of  actions  and  reactions 
by  both  aircraft  and  ground  threat  system  operations  is  required. 


Where  one  tool  is  weak,  another  may  be  strong.  By  using  all  the 
tools,  an  EW  tester  can  do  a  more  complete  job  of  testing.  The  integrated 
methodology  is  shown  in  Figure  13-1.  EW  integrated  testing  can  be 
summarized  as: 


(1)  Initial  modeling  phase  for  sensitivity  analysis  and  test 
planning. 

(2)  Active  test  phases  at  hybrid  laboratory  simulator  and 
field  range  facilities. 

(3)  Test  data  reduction  and  analysis. 

(4)  Post-test  modeling  phase  repeating  the  first  step  using 
test  data  for  extrapolation. 

(5)  Force  effectiveness  modeling  and  analysis  phase  to  determine 
the  incremental  contribution  of  the  new  system  to  total 
force  effectiveness. 

13.3  TESTING  OF  C2  SYSTEMS 

13.3.1  Special  Considerations  When  Testing  C2  Systems 

The  purpose  of  a  C2  system  is  to  provide  a  commander  with  timely 
and  relevant  information  to  support  sound  decision  making.  A  variety 
of  problems  face  the  C2  system  tester.  However,  In  evaluating  command 
effectiveness,  it  is  difficult  to  separate  the  contribution  made  by  the 
C2  system  from  the  contribution  made  by  the  commander's  Innate,  cognitive 
processes.  To  assess  a  C2  system  in  its  operational  environment.  It 
must  be  connected  to  the  other  systems  with  which  it  would  normally 
operate,  making  traceability  of  test  results  difficult.  Additionally, 
modern  C2  systems  are  software  intensive  and  highly  interactive,  with 
complex  man-machine  interfaces.  Measuring  C2  system  effectiveness  thus 
requires  the  tester  to  use  properly  trained  user  troops  during  the  test 
and  to  closely  monitor  software  T&E.  C2  systems  of  the  Army,  Navy,  Air 
Force,  and  Marines  are  expected  to  interoperate  with  each  other  and  with 
those  of  the  NATO  forces;  hence,  the  tester  must  also  ensure  Inter- Service 
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Figure  13-1.  Integrated  EW  Testing  Approach 


and  NATO  compatibility  and  interoperability. 

13.3.2  C2  Test  Facilities 

Testing  of  C2  systems  will  have  to  rely  more  on  the  use  of  computer 
simulations  and  C3I  test  beds  to  assess  their  overall  effectiveness. 
The  Joint  Tactical  Command,  Control,  and  Communications  Agency  (JTC3A) 
which  is  responsible  for  ensuring  interoperability  among  all  U.S.  tactical 
C3  systems  that  would  be  used  in  joint  or  combined  operations,  operates 
the  Joint  Test  Element  (JTE)  in  Ft.  Huachuca,  Arizona.  The  JTE  is  a 
testbed  for  C3I  systems  interoperability.  Another  facility,  the  huge 
testbed  developed  at  Kirtland  AFB,  New  Mexico  for  the  Identification 
Friend,  Foe,  or  Neutral  (IFFN)  Joint  Test,  will  be  operated  by  the  Air 
Force  and  be  available  for  use  by  the  development  and  operational 
communities  of  all  the  Services  for  their  C3I  testing  needs. 

13.4  CURRENT  TRENDS  IN  TESTING  C2  SYSTEMS 

13.4.1  Evolutionary  Acquisition  of  C2  Systems 

Evolutionary  Acquisition  (EA)  is  a  strategy  designed  to  provide 
an  early,  useful  capability  even  thougn  detailed  overall  system 
requirements  cannot  be  fully  defined  at  the  program's  inception.  EA 
contributes  to  a  reduction  in  the  risks  involved  in  system  acquisition, 
since  the  system  is  developed  and  tested  in  manageable  increments.  C2 
systems  are  likely  candidates  for  EA  because  they  are  characterized  by 
system  requirements  which  are  difficult  to  quantify  or  even  articulate 
and  which  are  expected  to  change  as  a  function  of  scenario,  mission, 
theater,  threat,  and  emerging  technology.  Therefore,  the  risk  associated 
with  developing  these  systems  can  be  very  great. 


Recent  studies  by  the  Defense  Systems  Management  College  and 
the  International  Test  and  Evaluation  Association  (ITEA)  have  addressed 
the  issues  involved  in  the  evolutionary  acquisition  and  testing  of  C2 
systems.  The  ITEA  study  illustrated  EA  in  Figure  13-2,  and  stated  that: 


With  regard  to  the  tester's  role  in  EA,  the  study  group  concluded 
that  iterative  test  and  evaluation  is  essential  for  success 
in  an  evolutionary  acquisition.  The  tester  must  become  involved 
early  in  the  acquisition  process  and  contribute  throughout 
the  development  and  fielding  of  the  core  and  the  subsequent 
Increments....  The  testers  contribute  to  the  requirements 
process  through  feedback  of  test  results  to  the  user. . .and. . .must 
judge  the  ability  of  the  system  to  evolve  (Reference  4). 
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Figure  13-2.  The  Evolutionary  Acquisition  Process 
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The  testing  of  Evolutionary  Acquisition  (EA)  Systems  presents  the 
tester  with  a  unique  challenge  as  he  must  test  the  core  system  during 
fielding  and  the  first  increment  before  the  core  testing  is  completed. 
This  could  lead  to  a  situation  where  the  tester  has  three  or  four  tests 
ongoing  on  various  increments  of  the  same  system.  The  Program  Manager 
must  insist  that  the  testing  for  EA  systems  be  carefully  planned  to  ensure 
the  test  data  is  shared  by  all  and  there  is  a  minimum  of  repetition  or 
duplicity  in  testing. 

13.4.2  Radio  Vulnerability 

The  Radio  Vulnerability  Analysis  (RVAN)  Methodology  is  a  recently 
developed  methodology  for  assessing  the  anti-jam  capability  and  limitations 
of  radio  frequency  data  links  when  operating  in  a  hostile  electronic 
countermeasures  environment.  In  1983,  OSD  directed  the  Services  to  apply 
the  Data  Link  Vulnerability  Analysis  (DVAL)  methodology  to  all  new  data 
links  being  developed. 


The  purpose  of  the  DVAL  methodology  is  to  identify  and  quantify 
the  anti-jam  capabilities  and  vulnerabilities  of  an  RF  data  link  operating 
in  a  hostile  electronic  countermeasures  (ECM)  environment.  The  methodology 
is  applied  throughout  the  acquisition  process  and  permits  early 
identification  of  needed  design  modifications  to  reduce  identified  ECM 
vulnerabilities.  The  following  four  components  determine  a  data  link's 
EW  vulnerability: 

(1)  The  susceptibility  of  a  data  link,  i.e.,  the  receiver's 
performance  when  subjected  to  intentional  threat  ECM; 

(2)  The  interceptibil ity  of  the  data  link,  i.e.,  the  degree 
to  which  the  transmitter  could  be  intercepted  by  enemy 
intercept  equipment; 

(3)  The  accessibility  of  the  data  link,  i.e.,  the  likelihood 
that  a  threat  jammer  could  degrade  the  data  link’s 
performance;  and 

(4)  The  feasibility  that  the  enemy  would  intercept  and  jam 
the  data  link  and  successfully  degrade  its  performance. 

The  analyst  applying  the  DVAL  methodology  will  require  test  data,  and 
the  test  manager  of  the  C3I  system  of  which  the  data  link  is  a  component, 
will  be  required  to  provide  this  data. 

13.5  SUMMARY 

EW  systems  must  be  tested  under  conditions  representative  of 
the  dense  threat  signal  environments  in  which  they  will  operate.  C2 
systems  must  be  tested  in  representative  environments  where  their 
interaction 
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and  responsiveness  can  be  demonstrated.  The  solution  for  the  tester 
is  an  integrated  approach  using  a  combination  of  analytical  models, 
computer  simulations,  hybrid  laboratory  simulators  and  test  beds,  and 
actual  field  testing.  The  tester  must  understand  these  test  techniques 
and  resources  and  apply  them  in  EW  and  C2  test  and  evaluation. 
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CHAPTER  14 

LOGISTICS  INFRASTRUCTURE  T&E 


14.1  INTRODUCTION 

In  all  materiel  acquisition  programs,  the  Integrated  Logistics 
Support  (ILS)  effort  begins  in  the  mission  area  analysis  phase  prior 
to  program  initiation,  continues  throughout  the  entire  acquisition  cycle, 
and  extends  past  the  deployment  phase.  Logistics  testing  must  therefore 
extend  over  the  entire  acquisition  cycle  of  the  system  and  be  carefully 
planned  and  executed  to  ensure  the  readiness  and  supportability  of  the 
system.  This  chapter  covers  the  development  of  logistics  support  test 
requirements  and  the  conduct  of  supportability  assessments  to  ensure 
that  readiness  and  supportability  objectives  are  identified  and  achieved. 
The  importance  of  the  ILS  Manager's  participation  in  the  TEMP  development 
process  should  be  stressed.  He  must  ensure  the  ILS  T&E  objectives  are 
considered  and  that  adequate  resources  are  available  for  ILS  T&E. 

Integrated  Logistic  Support  (ILS)  is  defined  as  a  disciplined, 
unified,  and  iterative  approach  to  the  management  and  technical  activities 
necessary  to:  integrate  support  considerations  into  system  and  equipment 
design;  develop  support  requirements  that  are  related  consistently  to 
readiness  objectives,  to  design,  and  to  each  other;  acquire  the  required 
support;  provide  the  required  support  during  the  operational  phase  at 
minimum  cost  (DoD  5000.39). 

ILS  consists  of  ten  specific  components,  or  elements: 

(1)  Maintenance  planning 

(2)  Manpower  and  personnel 

(3)  Supply  support 

(4)  Support  equipment 

(5)  Technical  data 

(6)  Training  and  training  support 

(7)  Computer  resources  support 

(8)  Facilities 

(9)  Packaging,  handling,  storage,  and  transportation 

(10)  Design  interface 

14.2  PLANNING  FOR  ILS  T&E 
14.2.1  Objectives  of  ILS  T&E 

The  main  objective  of  ILS  test  and  evaluation  is  to  verify 
that  the  logistic  support  being  developed  for  the  materiel  system  is 
capable  of  meeting  the  required  objectives  for  both  peacetime  and  wartime 
employment.  ILS  T&E  consists  of  the  usual  DT&E  and  OT&E,  but  also  Includes 
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post-deployment  supportability  assessments.  The  formal  DT&E  and  OT&E 
begin  in  the  concept  exploration  phase  and  continue  into  the 
production/deployment  phase.  Figure  14-1,  which  appears  in  the  DSMC 
Integrated  Logistics  Support  Guide,  describes  the  specific  DT,  OT,  and 
supportability  assessment  objectives  for  each  acquisition  phase. 

14.2.2  Planning  Documentation  for  ILS  T&E 

14.2.2.1  Integrated  Logistic  Support  Plan 

The  ILS  Manager  for  a  materiel  acquisition  system  is  responsible 
for  developing  the  Integrated  Logistic  Support  Plan  (ILSP),  which  is 
the  primary  document  for  planning  and  implementing  the  support  of  the 
fielded  system.  It  is  initially  prepared  during  the  Concept  Exploration 
phase,  then  is  progressively  developed  in  more  detail  as  the  system  moves 
through  the  acquisition  phases.  Included  in  the  ILSP  is  identification 
of  the  specific  ILS  test  issues  related  to  the  individual  ILS  elements 
and  the  overall  system  support  and  readiness  objectives. 

The  ILS  Manager  is  assisted  throughout  the  system's  development 
by  the  Integrated  Logistics  Support  Management  Team  (ILSMT),  which  is 
formed  early  in  the  acquisition  cycle.  The  ILSMT  is  a 
coordination/advisory  group  comprised  of  personnel  from  the  Program 
Management  office,  the  using  command,  and  other  commands  concerned  with 
acquisition  activities  such  as  logistics,  testing,  and  training. 

14.2.2.2  Supportability  Assessment  Plan 

Based  upon  the  ILSP  objectives,  the  ILS  Manager,  in  conjunction 
with  the  system's  test  manager,  develops  the  Supportabil ity  Assessment 
Plan,  which  identifies  the  testing  approach  and  the  evaluation  criteria 
that  will  be  used  to  assess  the  supportabil ity-related  design  requirements 
(e.g.  reliability  and  maintainability)  and  adequacy  of  the  planned  logistic 
support  resources  for  the  materiel  system.  Development  of  the 
Supportability  Assessment  Plan  begins  in  t  a  Concept  Exploration  phase; 
the  plan  Is  then  updated  and  refined  in  each  successive  acquisition  phase. 
The  ILS  tester  applies  the  techniques  of  Logistic  Support  Analysis,  as 
described  in  MI  L- STD- 1388- 1A,  to  formulate  the  test  and  evaluation 
strategy,  establish  the  T&E  program  objectives  and  criteria  and  identify 
required  test  resources.  He  must  ensure  that  his  T&E  strategy  is  based 
upon  quantified  supportability  requirements  and  will  address  supportability 
Issues  which  have  a  high  degree  of  risk  associated  with  them.  He  must 
also  ensure  that  the  necessary  quantities  and  types  of  data  will  be 
collected  to  validate  the  various  T&E  objectives,  both  during  system 
development  and  after  deployment  of  the  system.  The  T&E  objectives  and 
criteria  must  provide  a  basis  upon  which  to  ensure  that  critical 
supportability  Issues  and  requirements  are  resolved  or  achieved  within 
acceptable  confidence  levels. 
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SOURCE:  DSMC,  INTEGRATED  LOGISTICS  SUPPORT  GUIDE,  MAY  1986. 


14.2.2.3  Test  and  Evaluation  Master  Plan  (TEMP) 

The  Program  Manager  must  include  ILS  T&E  information  in  the 
TEMP,  as  specified  in  DoD  5000.3-M-l.  The  input,  which  is  derived  from 
the  Supportability  Assessment  Plan  with  the  assistance  of  the  ILS  Manager 
and  the  tester,  includes  descriptions  of  required  operational  suitability, 
specific  plans  for  testing  logistics  supportability,  and  required  testing 
resources.  It  is  of  critical  importance  that  all  test  resources  required 
for  all  ILS  testing  (DT,  OT,  and  post-deployment  supportability)  be 
identified  in  the  TEMP,  because  the  TEMP  is  the  basis  upon  which  test 
resources  are  budgeted  and  allocated  for  testing. 

14.2.3  Planning  Guidelines  for  Logistic  T&E 

The  following  guidelines  for  ILS  T&E  are  extracted  from  the 
DSMC  ILS  Guide: 

(1)  Develop  a  test  strategy  for  each  ILS-related  objective. 
Ensure  that  OT&E  planning  encompasses  all  ILS  elements. 
The  general  ILS  objectives  shown  in  Figure  14-1  must  be 
translated  into  detailed  quantitative  and  qualitative 
requirements  for  each  acquisition  phase  and  each  T&E 
program. 

(2)  Incorporate  ILS  testing  requirements  (where  feasible) 
into  the  formal  DT&E/OT&E  plans. 

(3)  Identify  ILS  T&E  that  will  be  performed  outside  of  the 
normal  DT&E  and  OT&E.  Include  subsystems  that  require 
off-system  evaluation. 

(4)  Identify  all  required  resources,  to  include  test  articles 
and  logistic  support  items  for  both  formal  DT/OT  and 
separate  ILS  testing  (participate  with  test  planner). 

(5)  Ensure  establishment  of  an  operationally  realistic  test 
environment,  to  include  personnel  representative  of  those 
who  will  eventually  operate  and  maintain  the  fielded  system. 
These  personnel  should  be  trained  for  the  test  using 
prototypes  of  the  actual  training  courses  and  devices 
and  should  be  supplied  with  drafts  of  all  technical  manuals 
and  documentation  that  will  be  used  with  the  fielded  system. 

(6)  Ensure  planned  OT&E  will  provide  sufficient  data  on  high 
cost  and  high  maintenance  burden  items  (e.g.,  for  high 
cost  critical  spares,  early  test  results  can  be  used  to 
re-evaluate  selection). 
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(7)  Participate  early  and  effectively  in  the  TEMP  development 
process  to  ensure  the  TEMP  includes  critical  logistic 
T&E  and  omission  of  needed  ILS  test  funds  from  program 
and  budget  documents. 

(8)  Identify  the  planned  utilization  of  all  data  collected 
during  the  assessments  to  avoid  mismatch  of  data  collection 
and  information  requirements. 

Detailed  evaluation  criteria  for  each  of  the  ten  ILS  elements  listed 
above  are  presented  in  Department  of  the  Army  Pamphlet  700-50,  "Integrated 
Logistic  Support:  Developmental  Supportabil ity  Test  and  Evaluation  Guide." 

14.3  CONDUCTING  ILS  T&E 

14.3.1  Scope 

The  purposes  of  ILS  T&E  are  to  measure  the  supportabil  ity  of 
a  developing  system  throughout  the  acquisition  process;  to  identify 
supportabil ity  deficiencies  and  potential  corrections/improvements  as 
test  data  becomes  available;  and  to  assess  the  operational  effectiveness 
of  the  planned  support  system.  ILS  T&E  also  evaluates  the  system's 
operational  suitability  and  its  ability  to  achieve  planned  readiness 
objectives  for  the  system/equipment  being  developed.  Specific  ILS  T&E 
tasks  (as  prescribed  in  MIL-STD-1388-1A)  include: 

o  Analysis  of  test  results  to  verify  achievement  of  specified 
supportabil ity  requirements. 

o  Determination  of  improvements  in  supportabil ity  and 

supportabil ity- related  design  parameters  needed  for  the 
system  to  meet  established  goals  and  thresholds. 

o  Identification  of  areas  where  established  goals  and 

thresholds  have  not  been  demonstrated  within  acceptable 
confidence  levels. 

o  Development  of  corrections  for  identified  supportabil ity 

problems  such  as  modifications  to  hardware,  software, 
support  plans,  logistic  support  resources,  or  operational 
tactics. 

o  Projection  of  changes  in  costs,  readiness  and  logistic 

support  resources  due  to  implementation  of  corrections. 

o  Analysis  of  supportabil ity  data  from  the  deployed  system 

to  verify  achievement  of  the  established  goals  and 


14-5 


thresholds  and  where  operational  results  deviate  from 
projections,  determination  of  the  causes  and  corrective 
actions. 

ILS  T&E  may  consist  of  a  series  of  ILS  demonstrations  and 
assessments  which  are  usually  conducted  as  part  of  system  performance 
tests.  Special  end-item  equipment  tests  are  rarely  conducted  solely 
for  ILS  evaluation. 

14.3.2  T&E  of  System  Support  Package 

T&E  of  the  support  for  a  materiel  system  requires  a  system 
support  package  consisting  of  spares,  support  equipment,  technical 
documents  and  publications,  representative  personnel,  any  peculiar  support 
requirements,  and  the  test  article  itself;  in  short,  all  of  the  items 
that  would  eventually  be  required  when  the  system  is  operational.  This 
complete  support  package  must  be  at  the  test  site  before  the  test  is 
scheduled  to  begin.  Delays  in  the  availability  of  certain  support  items 
could  prevent  the  test  from  proceeding  on  schedule  (which  can  be  costly 
due  to  on-site  support  personnel  on  hold  or  tightly  scheduled  system 
ranges  and  expensive  test  resources  not  being  properly  utilized)  or  could 
result  in  the  test  proceeding  without  conducting  the  complete  evaluation 
of  the  support  system.  The  ILS  test  planner  must  ensure  that  the  required 
personnel  are  trained  and  available,  that  test  facility  scheduling  is 
flexible  enough  to  permit  normal  delays,  and  that  the  test  support  package 
is  on  site  on  time. 

14.3.3  Data  Collection  and  Analysis 

The  ILS  Manager  must  coordinate  with  the  testers  to  ensure 
that  the  methods  used  for  collection,  storage,  and  extraction  of  ILS 
T&E  data  are  compatible  with  those  used  in  testing  the  materiel  system. 
As  with  any  testing,  the  ILS  test  planning  must  ensure  that  all  required 
data  is  identified;  that  it  is  sufficient  to  evaluate  a  system's  readiness 
and  supportability;  and  that  plans  are  made  for  a  data  management  system 
that  is  capable  of  the  data  classification,  storage,  retrieval,  and 
reduction  necessary  for  statistical  analysis. 

14.3.4  Use  of  ILS  Test  Results 

The  emphasis  on  the  use  of  the  results  of  testing  changes  as 
the  program  moves  from  the  CE  Phase  to  post  deployment.  During  early 
phases  of  a  program,  the  evaluation  results  are  used  primarily  to  verify 
analysis  and  develop  future  projections.  As  the  program  moves  into  FSD 
and  hardware  becomes  available,  the  evaluation  addresses  design, 
particularly  the  reliability  and  maintainability  aspects,  training 
programs,  support  equipment  adequacy,  personnel  skills  and  availability, 
and  technical  publications. 
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The  ILS  Manager  must  make  the  Program  Manager  (PM)  aware  of 
the  impact  on  the  program  of  logistical  shortcomings  which  are  identified 
during  the  T&E  process.  The  PM  in  turn  must  ensure  that  the  solutions 
to  any  shortcomings  are  identified  and  reflected  in  the  revised 
specifications  and  that  the  revised  test  requirements  are  included  in 
the  updated  Test  and  Evaluation  Master  Plan  (TEMP)  as  the  program  proceeds 
through  the  various  acquisition  stages. 

14.4  LIMITATIONS  TO  ILS  T&E 

Concurrent  testing  or  tests  that  have  accelerated  schedules  frequently 
do  not  have  sufficient  test  articles,  equipment  or  hardware  to  achieve 
statistical  confidence  in  the  testing  conducted.  The  shortage  of  equipment 
is  often  the  reason  that  shelf  life  and  service  life  testing  is  cut  short, 
leaving  the  ILS  evaluator  with  insufficient  data  to  predict  future 
performance  of  the  test  item. 

14.5  SUMMARY 

Test  and  Evaluation  are  the  logisticians'  tools  for  measuring  the 
ability  of  the  planned  support  system  to  fulfill  the  materiel  system's 
readiness  and  supportability  objectives.  The  effectiveness  of  ILS  T&E 
is  based  upon  the  completeness  and  timeliness  of  the  planning  effort. 

ILS  T&E  requirements  must  be  an  integral  part  of  the  TEMP  to  ensure 
budgeting  and  scheduling  of  required  test  resources.  Data  requirements 
must  be  completely  identified,  with  adequate  plans  made  for  collection, 
storage,  retrieval,  and  reduction  of  test  data. 
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CHAPTER  15 

PRODUCTION  RELATED  TESTING  ACTIVITIES 

15.1  INTRODUCTION 

Most  of  the  T&E  discussed  in  this  guidebook  concerns  the  testing 
of  the  actual  weapon  or  system  being  developed.  But  the  Program  Manager 
(PM)  must  also  evaluate  production  related  test  activities  and  the 
production  process  itself.  This  chapter  describes  production  management 
and  the  production  process  testing  required  to  ensure  the  effectiveness 
of  the  manufacturing  process  and  the  producibility  of  the  system's  design. 
Normally,  the  DT  and  OT  organizations  are  not  involved  directly  in  this 
process.  Usually,  the  Manufacturing  and  Quality  Assurance  sections  of 
the  program  office,  along  with  the  Program  Manager,  oversee/perform  many 
of  these  functions. 

15.2  PRODUCTION  MANAGEMENT 

Production  management  is  defined  as  "the  effective  use  of 
resources  to  produce,  on  schedule,  the  required  number  of  end  items  that 
meet  specified  quality,  performance,  and  cost.  Production  management 
includes,  but  is  not  limited  to,  industrial  resource  analysis, 
producibility  assessment,  producibility  engineering  and  planning, 
production  engineering,  industrial  preparedness  planning,  post-production 
planning,  and  productivity  enhancement"  (DODD  4245.6).  Production 
management  begins  early  in  the  acquisition  process,  as  early  as  the  Concept 
Exploration  phase,  and  is  specifically  addressed  at  each  program  milestone 
decision  point.  For  instance,  during  the  Concept  Exploration/Definition 
phase  (CE),  production  feasibility,  costs,  and  risks  should  be  addressed. 
Prior  to  Milestone  I,  the  PM  must  conduct  an  industrial  resource  analysis 
(IRA)  to  determine  the  availability  of  production  resources  (e.g.,  capital, 
material,  manpower)  required  to  support  the  production  of  the  weapon 
system.  Based  upon  the  results  of  the  IRA,  critical  materials, 

deficiencies  in  the  U.S.  industrial  base,  and  requirements  for  new  or 
updated  manufacturing  technology  can  be  identified.  Analysis  of  the 
industrial  base  capacity  is  one  of  the  considerations  In  preparing  the 
System  Concept  Paper  for  the  Milestone  I  decision.  As  development 

proceeds,  the  manufacturing  strategy  is  developed,  and  detailed  plans 
are  made  for  the  production  phase.  Independent  producibility  assessments, 
conducted  In  preparation  for  the  transition  from  development  to  production, 
are  reviewed  at  the  Milestone  II  decision  point.  At  Milestone  II,  the 
Full-Scale  Development  decision,  the  producibility  of  the  system  design 
concept  Is  evaluated  to  verify  that  the  system  can  be  manufactured  In 
compliance  with  the  production  cost  and  the  industrial  base  goals  and 
thresholds. 

The  Milestone  III  Full  Rate  Production  decision  is  supported 
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by  an  assessment  of  the  readiness  of  the  system  to  enter  production. 
The  system  cannot  enter  the  Full-Rate  Production/Deployment  phase  until 
it  is  determined  that  the  principal  contractors  have  the  necessary 
resources  (i.e.,  physical,  financial,  and  managerial  capacity)  to  achieve 
the  cost  and  schedule  commitments  and  to  meet  both  peacetime  and 
mobilization  requirements  for  production  of  the  system.  The  method  of 
assessing  production  readiness  in  preparation  for  Milestone  III  is  the 
Production  Readiness  Review  (PRR),  which  is  conducted  by  the  PM  and  staff. 
An  independent  assessment  of  production  readiness  is  conducted  at  the 
same  time  by  the  DoD  Product  Engineering  Services  Office  (DPESO)  of  Deputy 
Director  Defense  Research  and  Engineering  (DDDR&E).  The  DPESO  reports 
its  findings  directly  to  the  Defense  Acquisition  Executive. 

15.3  PRODUCTION  READINESS  REVIEWS 

The  policy,  procedures,  and  guidelines  for  PRRs  are  delineated 
in  DODD  5000.38,  which  states  "the  objective  of  a  PRR  is  to  verify  that 
the  production  design,  planning,  and  associated  preparations  for  a  system 
have  progressed  to  the  point  where  a  production  commitment  can  be  made 
without  incurring  unacceptable  risks  of  breaching  thresholds  of  schedule, 
performance,  cost,  or  other  established  criteria."  The  PRR  must  confirm 
that  the  system  design  is  stable  and  producible;  that  adequate 
manufacturing  technology  exists;  that  the  necessary  manufacturing  methods, 
techniques,  and  processes  are  available  to  the  producer;  and  that  suitable 
provisions  have  been  made  for  manufacturing,  cost,  and  quality  control. 

The  conduct  of  a  PRR  is  the  responsibility  of  the  PM,  who  usually 
appoints  a  director.  The  director  then  assembles  a  team  made  up  of 
individuals  with  design,  industrial,  manufacturing,  procurement,  inventory 
control,  contracts,  the  engineering  disciplines,  and  quality  training 
experience.  The  PRR  director  organizes  and  manages  the  team  effort  and 
supervises  preparation  of  the  findings.  The  PRR  is  conducted  as  a 
time-phased  effort  during  the  Full-Scale  Development  phase  following 
the  guidelines  presented  in  DODD  5000.38.  Table  15-1  summarizes  these 
guidelines  in  a  checklist  format. 

15.4  QUALIFICATION  TESTING 

Qualification  Testing  is  performed  to  verify  the  design  and 
manufacturing  process,  and  it  provides  a  baseline  for  subsequent  acceptance 
tests.  The  production  qualification  testing  is  conducted  at  the  unit, 
sub-system,  and  system  level  on  production  items  and  is  completed  before 
the  production  decision.  The  results  of  these  tests  are  a  critical  factor 
in  assessing  the  system's  readiness  for  production.  There  are  normally 
preproduction  and  production  qualification  tests.  Downline  production 
qualification  tests  are  performed  to  verify  process  control  and  may  be 
performed  on  selected  parameters  rather  than  at  the  levels  originally 
selected  for  qualification. 
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15.4.1  Preproduction  Qualification  Tests  (PPQT) 

Preproduction  Qualification  Tests  are  a  series  of  formal 
contractual  tests  conducted  to  ensure  design  integrity  over  the  specified 
operational  and  environmental  range.  The  tests  are  conducted  on  prototype 
or  preproduction  items  fabricated  to  the  proposed  production  design 
drawings  and  specifications.  The  PPQTs  include  all  contractu?'  Tity 
and  maintainability  demonstration  tests  required  prior  to  j.uwtion 
release.  For  volume  acquisitions,  these  tests  are  a  constraint  to 
production  release. 


Table  15-1.  PRR  Guidelines  Checklist 


Product  Design 

•  Producible  at  low  risk 

•  Stabilized  at  low  rate  of  change 

•  Validated 

•  Reliability,  maintainability,  performance  demonstrated 

•  Components  engineering  has  approved  all  parts  selections 

Industrial  Resources 

•  Adequate  plant  capacity  (peacetime  and  wartime  demands) 

•  Facilities,  special  production  and  test  equipment, 
tooling  identified 

•  Needed  plant  modernization  (CAD/CAM,  other  automation) 

accomplished 

which  produces  an  invested  captive  payback  in  two  to  five  years 

•  Associated  computer  software  developed 

•  Skilled  personnel  and  training  programs  available 

Production  Engineering  and  Planning 

•  Production  plan  developed  (reference  MIL-STD-152 8 ) 

•  Production  schedules  compatible  with  delivery  requirements 

•  Manufacturing  methods  and  processes  integrated  with  facilities, 
equipment,  tooling,  and  plant  layout 

•  Value  engineering  applied 

•  Alternate  production  approaches  available 

•  Drawings,  standards,  and  shop  instructions  are  explicit 

•  Configuration  management  adequate 

•  Production  policies  and  procedures  documented 

•  Management  information  system  adequate 

•  Contractor's  management  structure  is  acceptable  to  the  PMO 

•  The  PEP  checklist  has  been  reviewed 
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Table  15-1.  PRR  Guidelines  Checklist  (Concluded) 


Materials 

•  All  selected  materiels  approved  by  contractor's 
Materiel  Engineers 

•  Bill  of  materials  prepared 

•  "Make-or-Buy"  decisions  complete 

•  Procurement  of  long  lead-time  items  planned 

•  Sole  source  and  government  furnished  items  identified 

•  Contractor's  inventory  control  system  adequate 

•  Contractor's  material  cost  procurement  plan  complete 

Quality  Assurance 

•  Quality  plan  in  accordance  wht  contract  requirements 

•  Quality  control  procedures  and  acceptance  criteria  established 

•  QA  organization  participates  in  production  planning  effort 

Laa.ia.ti.sa 

•  Operational  support,  tes  and  diagnostic  equipment  available 
at  system  deployment 

•  Training  aids,  simulators,  other  devices  ready  at  system 
deployment 

•  Spares  integrated  into  production  lot  flow 


15.4.2  Production  Qualification  and  Production  Acceptance  Tests 

Production  Qualification  and  Production  Acceptance  Tests  consist 
of  a  series  of  formal  contractual  tests  conducted  to  ensure  the 
effectiveness  of  the  manufacturing  process,  equipment,  and  procedures. 
These  t^sts  are  conducted  on  a  random  sample  from  the  first  production 
lot.  These  series  of  tests  are  repeated  if  the  manufacturing  process, 
equipment,  or  procedures  are  changed  significantly  and  when  a  second 
or  alternative  source  of  manufacturing  is  brought  on  line. 

15.5  TRANSITION  TO  PRODUCTION 

In  an  acquisition  process,  often  the  first  indications  that 
a  system  will  experience  problems  Is  during  the  transition  from  Full 
Scale  Development  to  Full-Rate  Production/Deployment.  This  transition 
continue  ove*'  an  extended  period,  often  months  or  years,  and  this  is 
the  period  that  the  system  Is  undergoing  stringent  contractor  and 
Government  testing.  There  may  be  unexpected  failures  that  require 
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significant  design  changes,  and  these  changes  may  impact  on  quality, 
producibility,  supportabil ity,  and  may  require  program  schedule  slippages. 

Long  periods  of  transition  usually  indicate  that  insufficient 
attention  to  design  or  producibility  needs  were  not  made  as  early  as 
Combat  Exploration  or  the  Demonstration/Validation  phases. 

15.5.1  The  Transition  Plan 

The  Transition  Plan  is  the  common  thread  that  guides  a  system 
from  CE  to  production.  The  plan  is  a  management  tool  which  ensures  that 
adequate  risk  handling  measures  have  been  taken  to  transition  from 
development  to  production.  It  contains  a  checklist  to  be  used  during 
the  readiness  reviews.  The  plan  should  tie  together  the  applications 
of  design,  test,  and  manufacturing  activities  in  order  to  reduce  data 

requirements,  duplication  of  effort,  cost  and  schedule,  and  assure  an 
early  success  of  the  FSD  first  production  article. 

15.5.2  Testing  During  the  Transition 

Testing  accomplished  during  the  transition  from  development 
to  production  will  include  the  Acceptance  Testing,  Manufacturing  Screening, 
and  Final  testing.  These  technical  tests  are  performed  by  the  contractor 
to  ensure  that  the  system  will  transition  smoothly,  and  that  test  design 
and  manufacturing  issues  affecting  design  and  design  issues  are  addressed. 
During  this  same  period,  the  Government  will  be  conducting  the  Initial 
Operational  Test  and  Evaluation,  which  can  be  conducted  in  two  phases 

during  the  transition,  using  the  latest  available  configuration  item. 
The  impact  of  these  tests  may  overwhelm  the  configuration  management 

of  the  system  unless  careful  planning  is  accomplished  to  handle  these 
changes. 

15.6  LOW  RATE  INITIAL  PRODUCTION  (LRIP) 

Systems  may  be  produced  in  a  limited  quantity  to  be  used  in 

OT&E  for  verification  of  production  engineering;  and  design  maturity 
and  to  establish  a  production  base.  Test  and  evaluation  is  conducted 
on  these  systems  to  verify  that  the  production  process  provides  materiel 
that  meets  the  required  technical  and  operational  performance  requirements 
of  the  system.  When  the  decision  authority  thinks  that  the  systems  will 
not  perform  to  expectation,  he  will  direct  that  it  not  proceed  beyond 
low-rate  initial  production.  The  Director,  Operational  Test  &  Evaluation 
submits  a  report,  on  all  major  systems,  to  Congressional  Committees  before 
the  Milestone  III  decision  to  proceed  beyond  L3IP  is  made. 

15.7  PRODUCTION  ACCEPTANCE  TEST  &  EVALUATION  (PAT&E) 

Production  Acceptance  Test  &  Evaluation  assures  that  production 
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items  demonstrate  the  fulfillment  of  the  requirements  and  specifications 
of  the  procuring  contract  or  agreements.  The  testing  also  ensures  the 
system  being  produced  demonstrates  the  same  performance  as  the 
preproduction  models.  The  procured  items  or  system  must  operate  in 
accordance  with  Type  A  (system)  and  Type  C  (production)  specifications. 
PAT&E  is  usually  conducted  by  the  Quality  Assurance  section  of  the  program 
office  at  the  contractor's  plant  and  may  involve  operational  users. 

For  example,  for  the  Rockwell  B-1B  Bomber  production 
acceptance,  both  Rockwell  and  Air  Force  Quality 
Assurance  inspectors  reviewed  all  manufacturing 
and  ground  testing  results  for  each  aircraft.  In 
addition,  a  flight  test  team  composed  of  both 
contractor  and  Air  Force  test  pilots  flew  each 
aircraft  a  minimum  of  10  hours,  demonstrating  in 
flight  all  on-board  aircraft  systems.  Any 

discrepancies  in  flight  were  noted,  corrected  and 
tested  on  the  ground,  then  retested  on  subsequent 
checkout  and  acceptance  flights.  Once  each  aircraft 
had  passed  all  tests  and  all  systems  were  fully 
operational.  Air  Force  authorities  accepted  the 
aircraft.  The  test  documentation  also  became  part 
of  the  delivered  package.  During  this  test  period, 
the  program  office  monitored  each  aircraft's  daily 
progress. 

15.8  SUMMARY 

A  primary  purpose  of  production  related  testing  is  to 
lower  the  production  risk  in  a  major  defense  acquisition  program.  The 
Program  Manager  must  ensure  that  the  contractor's  manufacturing  strategy 
and  capabilities  will  result  in  the  desired  product  within  acceptable 
cost.  LRIP  and  PAT&E  also  play  major  roles  in  ensuring  the  production 
unit  is  identical  to  the  preproduction  units  and  conforms  to  the 
specifications  of  the  contract. 
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MODULE  V 


TEST  AND  EVALUATION  ISSUES 


Many  Program  Managers  face  several  test  and  evaluation  issues  that 
must  be  resolved  to  get  their  particular  weapon  system  tested  and 
ultimately  fielded.  These  issues  may  include  modeling  and  simulation 
support;  combined  and  concurrent  testing;  test  resources;  survivability 
and  lethality  testing;  multiservice  testing;  or  international  T&E .  Each 
of  these  issues  presents  a  unique  set  of  challenges  for  the  Program 
Manager. 


CHAPTER  16 

MODELING  AND  SIMULATION  SUPPORT  TO  T&E 


16.1  INTRODUCTION 

This  chapter  discusses  the  applications  of  modeling  and 
simulation  in  test  and  evaluation.  The  need  for  modeling  and  simulation 
has  long  been  recognized,  as  evidenced  by  this  quote  from  the  USAF 
Scientific  Advisory  Board  in  June  1965: 

Prediction  of  combat  effectiveness  can  only  be,  and 
therefore  must  be,  made  by  using  the  test  data  in  analytical 
procedures.  This  analysis  usually  involves  some  type 
of  model,  simulation,  or  game  ( i . e . ,  the  tools  of  operations 
or  research  analysis).  It  is  the  exception  and  rarely, 
that  the  'end  result'  i.e.,  combat  effectiveness,  can 
be  deduced  directly  from  test  measurements. 

DoD  5000.3,  in  mandating  T&E  early  in  the  acquisition  process 
(i.e.,  prior  to  Milestone  II),  encourages  the  use  of  modeling  and 
simulation  as  a  source  of  T&E  data.  For  instance,  the  Armored  Family 
of  Vehicles  program  is  using  more  than  sixty  models,  simulations  and 
other  test  data  to  support  system  concept  exploration.  The  reliance 
on  modeling  and  simulation  by  this  and  other  acquisition  programs  provides 
the  T&E  community  with  valuable  information  which  can  increase  confidence 
levels,  decrease  field  test  time  and  costs,  and  provide  data  for  pre-test 
prediction  and  post-test  validation. 

This  chapter  demonstrates  that  proper  selection,  application, 
and  use  of  modeling  and  simulation  can  increase  the  efficiency  of  the 
T&E  process,  reduce  the  time  and  cost,  provide  otherwise  unattainable 
and  unmeasurable  data,  and  provide  more  timely  and  valid  results. 

16.2  TYPES  OF  MODELS  AND  SIMULATIONS 

The  term  "modeling  and  simulation"  is  often  associated  with 
huge  digital  computer  simulations,  but  it  also  includes  manual  and  man-in- 
the-loop  war  games,  test  beds,  hybrid  laboratory  simulators,  and 
prototypes. 

A  mathematical  model  is  an  abstract  representation  of  a  system 
that  provides  a  means  of  developing  quantitative  performance  requirements 
from  which  candidate  designs  can  be  developed.  Static  models  are  those 
that  depict  conditions  of  state,  while  dynamic  models  depict  conditions 
that  vary  with  time,  such  as  the  action  of  an  autopilot  in  controlling 
an  aircraft.  Simple  dynamic  models  can  be  solved  analytically,  and  the 
results  represented  graphically. 
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According  to  a  former  Director,  Defense  Test  and  Evaluation, 
(Reference  121),  simulations  used  in  T&E  can  be  divided  into  three 

categories:  computer  simulations,  system  test  beds,  and  system  prototypes. 
Computer  simulations  are  strictly  mathematical  representations  of  systems 
and  do  not  employ  any  actual  hardware.  They  may,  however,  incorporate 
some  of  the  actual  software  that  might  be  used  in  a  system.  Early  in 

a  system's  life  cycle,  computer  simulations  can  be  expected  to  provide 
the  most  system  evaluation  information.  In  many  cases,  computer 

simulations  can  be  readily  developed  as  modifications  of  existing 

simulations  for  similar  systems.  For  example,  successive  generations 
of  AIM-7  missile  simulations  have  been  effectively  used  in  test  and 

evaluation. 

A  system  test  bed  usually  differs  from  a  computer  simulation 
in  that  it  contains  some,  but  not  necessarily  all,  of  the  actual  hardware 
that  will  be  a  part  of  the  system.  Other  elements  of  the  system  are 
either  not  incorporated  or  are  incorporated  in  the  form  of  computer 
simulations.  The  system  operating  environment  (including  threat)  may 
either  be  physically  simulated,  as  in  the  case  of  a  flying  test  bed, 

or  it  may  be  computer  simulated,  as  in  the  case  of  a  laboratory  test 
bed.  Aircraft  cockpit  simulators  used  to  evaluate  pilot  performance 
are  good  examples  of  system  test  beds.  As  development  of  a  system 

progresses,  more  and  more  subsystems  become  available  in  hardware  form. 
These  subsystems  can  then  be  incorporated  into  system  test  beds  that 

typically  provide  a  great  deal  of  the  system  evaluation  information  used 
during  the  middle  part  of  a  system's  development  cycle. 

The  third  type  of  simulation  used  in  test  and  evaluation  is 
the  system  prototype.  Unlike  the  system  test  bed,  all  subsystems  are 
physically  incorporated  in  a  system  prototype.  The  system  prototype 
may  come  reasonably  close  to  representing  the  final  system  configuration 
depending  on  the  state  of  development  of  the  various  subsystems  which 
compose  it.  Preproduction  prototype  missiles  and  aircraft  used  in 
operational  testing  by  the  Services  are  examples  of  this  class  of 
simulation.  As  system  development  proceeds,  eventually  all  subsystems 
will  become  available  for  incorporation  in  one  or  more  system  prototypes. 
Operational  testing  of  these  prototypes  frequently  provides  much  of  the 
system  evaluation  Information  needed  for  a  decision  on  full  scale 
production  and  deployment. 

There  is  a  continuous  spectrum  of  simulation  types,  as  illustrated 
In  Figure  16-1,  with  the  pure  computer  simulation  at  one  end  and  the 
pure  hardware  prototype  at  the  other  end. 

16.3  VALIDITY  OF  MODELING  AND  SIMULATION 

Simulations  are  not  a  substitute  for  live  testing.  There  are 
many  things  that  cannot  be  adequately  simulated  by  computer  programs 
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THE  SIMULATION  SPECTRUM 


-  among  them  the  process  of  decision  and  the  proficiency  of  personnel 
in  the  performance  of  their  functions.  Therefore,  there  is  no  possible 
substitution  for  physical  tests  and  evaluations.  Simulations,  both  manual 
and  computer  designed,  however,  can  complement  and  increase  the  validity 
of  live  tests  and  evaluations  by  proper  selection  and  application.  Figure 
16-2  contrasts  the  test  criteria  which  are  conducive  to  modeling  and 
simulation  versus  physical  testing.  Careful  selection  of  the  simulation, 
knowledge  of  its  application  and  operation,  and  meticulous  selection 
of  input  data  will  produce  representative  and  valid  results. 

The  important  element  in  using  a  simulation  is  to  select  one 
that  is  representative  and  either  addresses  or  is  capable  of  being  modified 
to  address  the  level  of  detail  (the  essential  elements  of  analysis  and 
objectives)  under  investigation. 

16.4  SUPPORT  TO  TEST  DESIGN  AND  PLANNING 

Modeling  and  simulation  can  assist  in  the  T&E  planning  process 
and  can  reduce  the  cost  of  the  conduct  of  testing.  Areas  of  particular 
application  include  scenario  development  and  the  timing  of  test  events; 
the  development  of  objectives,  essential  elements  of  analysis,  and  measures 
of  effectiveness;  the  identification  of  variables  for  control  and 
measurement,  and  the  development  of  data  collection,  instrumentation 
and  data  analysis  plans.  For  example,  using  simulation,  the  test  designer 
can  examine  system  sensitivities  to  changes  in  variables  to  determine 
the  critical  variables  and  their  ranges  of  values  to  be  tested.  He  can 
also  predict  ahead  of  time  the  effects  of  various  assumptions  and 
constraints  and  evaluate  candidate  measures  of  effectiveness  to  help 
in  formulation  of  the  test  design. 

Caution  must  be  exercised  when  planning  to  rely  on  simulations 
as  a  means  of  obtaining  test  data  as  they  tend  to  be  very  expensive  to 
develop,  difficult  to  integrate  with  data  from  other  sources,  and  often 
do  not  provide  the  level  of  realism  required  for  operational  tests. 
Although  simulations  are  not  a  "cure  all"  they  should  be  used  whenever 
feasible  as  another  source  of  data  for  the  evaluator  to  consider  during 
the  test  evaluation. 

Computer  simulations  may  be  used  to  test  the  planning  for  an 
exercise.  By  setting  up  and  running  the  test  exercise  in  a  simulation, 
the  timing  and  scenario  may  be  tested  and  validated.  Critical  events 
may  be  identified  to  include  the  interaction  of  the  various  forces  which 
test  the  measures  of  effectiveness,  the  essential  elements  of  analysis 
and,  in  turn,  the  test  objectives.  Further,  the  simulation  may  be  used 
to  verify  the  statistical  test  design,  the  instrumentation  plan,  the 
data  collection  plan,  and  the  data  analysis  plan.  Essentially,  the  purpose 
of  the  computer  simulation  in  pre-test  planning  is  to  pre-test  the  test 
to  make  test  results  more  effective.  Pre-testing  attempts  to  optimize 
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Figure  16-2.  Values  of  Selected  Criteria  Conducive  to  Modeling  and  Simulation 
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test  results  by  pointing  out  potential  trouble  spots.  It  constitutes 
a  test  setup  analysis  which  can  encompass  a  multitude  of  areas. 


As  an  example  of  simulations  used  in  test  planning,  consider 

a  model  which  portrays  aircraft  versus  air  defenses.  The  model  can  be 
used  to  replicate  typical  scenarios  and  provide  data  on  the  number  of 
engagements,  the  air  defense  systems  involved,  the  aircraft  target,  the 
length  and  quality  of  the  engagement  and  a  rough  approximation  of  the 
success  of  the  mission  ( i . e . ,  did  the  aircraft  make  it  to  the  target?). 
With  such  data  available,  a  data  collection  plan  can  be  developed  to 
specify  in  more  detail  when  and  where  data  should  be  collected,  from 

which  systems,  and  in  what  quantity.  The  results  of  this  analysis  impact 
heavily  on  long  lead  time  items  such  as  data  collection  devices  and  data 
processing  systems.  The  more  specificity  available,  the  fewer  the  number 
of  surprises  which  will  occur  downstream.  As  tactics  are  decided  upon 
and  typical  flight  paths  are  generated  for  the  scenario,  an  analysis 

can  be  done  on  the  flight  paths  over  the  terrain  in  question  and  a 

determination  can  be  made  whether  or  not  the  existing  instrumentation 
can  track  the  numbers  of  aircraft  involved  in  their  maneuvering  envelopes. 
Alternative  siting  arrangements  can  be  examined  and  trade-offs  can  be 

made  between  the  amount  of  equipment  to  be  purchased  and  the  types  of 
profiles  which  can  be  tracked  for  this  particular  test.  Use  of  such 

a  model  can  also  highlight  numerous  choices  available  to  the  threat  air 
defense  system  in  terms  of  opportunities  for  engagement,  and  practical 
applications  of  doctrine  to  the  specific  situations. 

16.5  SUPPORT  TO  TEST  EXECUTION 

Simulations  can  be  useful  in  test  execution  and  dynamic  planning. 
With  funds  and  other  restrictions  limiting  the  number  of  times  that  a 
test  may  be  repeated  and  each  test  conducted  over  several  days,  it  is 

mandatory  that  the  test  director  exercise  close  control  over  the  conduct 
of  the  test  to  ensure  that  data  to  meet  the  test  objectives  are  being 
gathered  and  to  ensure  adequate  safety.  He  must  be  able  to  make  minor 
modifications  to  the  test  plan  and  scenario  to  force  achievement  of  these 
goals.  This  calls  for  a  dynamic  (quick-look)  analysis  capability  and 
a  dynamic  planning  capability.  Simulations  may  contribute  to  this 
capability.  For  example,  using  the  same  simulation(s)  as  used  in  the 

pre-test  planning,  the  tester  could  input  data  gathered  during  the  first 
day  of  the  exercise  to  determine  the  adequacy  of  the  data  to  fulfill 
the  test  objectives.  Using  this  data,  the  entire  test  could  be  simulated. 
Projected  inadequacies  could  be  isolated  and  the  test  plans  modified 
to  minimize  the  deficiencies. 

Simulations  may  also  be  used  to  support  test  control  and  to 
assure  safety.  For  example,  during  recent  missile  test  firings  at  White 
Sands  Missile  Range  (WSMR),  aerodynamic  simulations  of  the  proposed  test 
were  run  on  a  computer  during  actual  firings  so  that  real  time  missile 
position  data  could  be  continuously  compared  to  the  simulated 
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missile  position  data.  If  any  significant  variations  occurred,  and  if 
the  range  safety  officer  were  too  slow  (both  types  of  position  data  were 
displayed  on  plotting  boards),  the  computer  issued  a  destruct  command. 

Simulations  can  be  used  to  augment  tests  by  simulating 
non-testable  events  and  scenarios.  Although  operational  testing  should 
be  accomplished  in  as  realistic  an  operational  environment  as  possible, 
pragmatically  some  environments  are  impossible  to  simulate  for  safety 
or  other  reasons.  Some  of  these  include  the  environment  of  a  nuclear 
battlefield  to  include  the  effects  of  nuclear  bursts  on  both  friendly 
and  enemy  elements.  Others  include  two-sided  live  firings  and  adequate 
representation  of  other  forces  to  ascertain  compatibility  and 
interoperabil ity  data.  Instrumentation,  data  collection,  and  data 
reduction  of  large  combined  arms  forces  (e.g.,  brigade,  division,  and 
larger-sized  forces)  become  extremely  difficult  and  costly.  Simulations 
are  not  restricted  by  safety  factors  and  can  realistically  play  many 
of  the  environments  that  are  otherwise  unachievable  in  an  OT&E  -  nuclear 
effects,  large  combined  forces,  ECM  and  ECCM,  and  many-on-many  engagements. 

Usually,  insufficient  units  are  available  to  simulate  the 
organizational  relationships  and  interaction  of  the  equipment  with  its 
operational  environment,  particularly  during  the  early  OT&E  conducted 
using  prototype  or  pilot  production  type  equipment.  Simulations  are 
not  constrained  by  these  limitations.  Data  obtained  from  a  limited  test 
can  be  plugged  into  a  simulation  which  is  capable  of  handling  many  of 
the  types  of  equipment  being  tested  and  interfacing  them  with  other 
elements  of  the  blue  forces  and  operating  them  against  large  elements 
of  the  red  forces  to  obtain  interactions. 

Simulations  can  also  play  design  characteristics  of  equipment 
and  can  be  used  to  augment  the  results  obtained  using  prototype  or  pilot 
production  type  equipment  that  is  "mocked-up"  to  represent  the  final 
item.  The  simulation  may  be  used  to  improve  the  "mocked-up"  representation 
or  to  indicate  that  the  prototype  equipment  will  not  satisfy  the 
requirements  of  the  test. 

It  is  often  necessary  to  use  "substitute"  equipment  in  testing 
e.g.,  American  equipment  is  used  to  represent  threat  force  equipment. 
In  some  cases  the  substitute  equipment  may  have  greater  capabilities 
than  the  real  equipment;  in  other  cases,  less.  Simulations  are  capable 
of  representing  the  real  characteristics  of  equipments  and,  therefore, 
can  be  used  as  a  means  of  modifying  raw  data  collected  during  the  test 
to  reflect  real  characteristics. 

As  an  example,  suppose  the  substitute  equipment 

is  an  AAA  gun  with  a  tracking  ra'.e  of  30  degrees 

per  second,  whereas  the  equipment  for  which  it 
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is  substituted  has  a  tracking  rate  of  45  degrees 
per  second.  The  computer  simulation  could  be  used 
to  augment  the  collected,  measured  data  by  determining 
how  many  rounds  could  have  actually  been  fired  against 
each  target  or  whether  targets  that  were  missed 
because  of  too  slow  tracking  rate  could  have  been 
engaged  by  the  actual  equipment.  Consideration 
of  other  differing  factors  simultaneously  could 
have  a  plus  or  minus  synergistic  effect  on  test 
results. 

16.6  SUPPORT  TO  ANALYSIS  AND  TEST  REPORTING 


Modeling  and  simulation  may  be  used  in  post-test  analysis 
to  extend  and  generalize  results  and  to  extrapolate  to  other  conditions. 
The  difficulty  of  controlling  large  exercises,  not  to  mention  the 
difficulty  in  instrumenting  them  and  collecting  and  reducing  the  data, 
to  some  degree  limits  the  size  of  OT&E.  This  makes  the  process  of 
determining  the  operational  suitability  of  equipment  to  include 
compatibility,  interoperability,  organization,  etc.,  a  difficult  one. 
To  a  large  degree  the  interactions,  interrelationships,  and  compatibility 
of  large  forces  may  be  obtained  by  using  actual  data  collected  during 
the  test  and  applying  it  to  the  simulation. 

Simulations  can  be  used  to  extend  test  results  and  to 
save  considerable  energy  (both  fuel  and  manpower)  and  money  by  precluding 
the  need  to  rerun  to  improve  the  statistical  sample,  or  to  determine 
overlooked  or  directly  unmeasured  parameters. 

In  analyzing  the  test  results,  they  can  be  compared  to 

the  results  predicted  by  the  simulations  used  early  in  the  planning 
process.  Thus,  the  simulation  is  validated  by  the  actual  live  test 
results,  but  the  test  results  are  also  validated  by  the  simulation. 

16.7  SUtWARY 

Modeling  and  simulation  in  T&E  can  be  used  for  concept 

evaluation,  extrapolation.  Isolation  of  effects,  efficiency,  representation 
of  complex  environments,  and  overcoming  inherent  limitations  in  live 
testing.  The  use  of  modeling  and  simulation  can  validate  test  results. 
Increase  confidence  levels,  reduce  test  costs,  and  shorten  the  overall 

acquisition  cycle  by  providing  more  data  earlier  for  the  decision  maker. 
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MODULE  VI 


MANAGEMENT  OF  TEST  AND  EVALUATION 


Test  and  Evaluation  is  a  management  tool  and  an  integral  part  of 
the  development  process.  This  module  will  address  the  policy  structure 
and  oversight  mechanisms  in  place  for  test  &  evaluation.  It  will  also 
address  the  program  office  responsibilities  for  development  test  & 
evaluation  and  for  operational  test  &  evaluation.  The  module  will  con¬ 
clude  with  the  T&E  of  certain  weapon  system  types  and  outline  the  unique 
character  and  planning  requirements  of  each. 

The  module  will  also  address  the  T&E  from  an  international  per¬ 
spective  and  will  describe  the  OSD-sponsored  program  to  manage  the  test¬ 
ing  of  foreign  weapon  systems . 


CHAPTER  17 

COMBINED  AND  CONCURRENT  TESTING 


17.1  INTRODUCTION 

The  terms  "concurrency,"  "concurrent  testing,"  and  "combined 
testing"  are  sometimes  subject  to  misinterpretation.  For  the  purpose 
of  this  discussion,  "concurrency"  is  defined  as  an  approach  to  system 
development  and  acquisition  in  which  two  phases  of  the  acquisition  process, 
which  normally  occur  sequentially,  overlap  to  some  extent.  For  example, 
a  weapon  system  enters  the  production  phase  while  development  efforts 
are  still  underway. 

The  term  "concurrent  testing"  refers  to  circumstances  when 
development  testing  and  operational  testing  take  place  at  the  same  time, 
as  two  parallel  but  clearly  separate  and  distinct  activities.  In  contrast, 
"combined  testing"  refers  to  a  single  test  program  conducted  to  support 
both  development  test  and  operational  test  objectives.  This  chapter 
discusses  the  use  of  combined  testing  and  concurrent  testing  and  highlights 
some  of  the  advantages  and  disadvantages  associated  with  these  approaches. 

17.2  COMBINING  DT  AND  OT 

Certain  test  events  can  be  organized  to  provide  information 
that  is  useful  to  both  development  testers  and  operational  testers. 
For  example,  a  prototype  free-fall  munition  could  be  released  from  a 
fighter  aircraft  at  operational  employment  conditions  to  satisfy  both 
development  and  operational  test  objectives.  Such  instances  need  to 
be  identified  to  prevent  unnecessary  duplication  of  effort  and  to  control 
costs.  A  combined  testing  approach  is  also  appropriate  for  certain 
specialized  types  of  testing.  For  example,  in  the  case  of  nuclear 
survivability  and  hardness  testing,  systems  cannot  be  tested  in  a  totally 
realistic  operational  environment,  therefore,  a  single  test  program  is 
often  used  to  meet  both  development  and  operational  test  objectives. 

DoD  Directive  5000.3  encourages  combined  testing  and  states 
that  "a  combined  DT&E  and  OT&E  approach  may  be  used  when  cost  and  time 
benefits  are  significant  and  are  clearly  identified,  provided  that  test 
objectives  are  not  compromised."  If  this  approach  is  elected,  planning 
efforts  must  be  carefully  coordinated  early  in  the  program  to  ensure 
that  data  is  obtained  to  satisfy  the  needs  of  both  the  developing  agency 
and  the  independent  operational  tester.  Care  must  also  be  exercised 
to  ensure  that  a  combined  test  program  contains  dedicated  operational 
test  events  to  satisfy  the  requirement  for  an  independent  evaluation. 
The  final  period  of  testing  before  the  full -rate  production  decision 
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will  be  separate  OT&E  managed  by  the  independent  operational  test  agency. 
In  all  combined  test  programs,  separate  independent  development  and 
operational  evaluations  of  test  results  must  be  provided. 

Service  regulations  describe  the  sequence  of  activities  in 
a  combined  testing  program  as  follows: 

Although  IOT&E  is  separate  and  distinct  from  DT&E, 
most  of  the  generated  data  are  mutually  beneficial 
and  freely  shared.  Similarly,  the  resources  needed 
to  conduct  and  support  both  test  efforts  are  often 
the  same  or  very  similar.  Thus,  when  sequential 
DT&E  and  IOT&E  efforts  would  cause  delay  or  increase 
the  acquisition  cost  of  the  system,  DT&E  and  IOT&E 
are  combined.  When  combined  testing  is  planned, 
the  necessary  test  conditions  and  data  required  by 
both  DT&E  and  OT&E  organizations  must  be  integrated. 

Combined  testing  can  normally  be  divided  into  three 
segments.  In  the  first  segment,  DT&E  event[s]  usually 
assume  priority  because  critical  technical  and 
engineering  tests  must  be  accomplished  to  continue 
the  engineering  and  development  process.  During 
this  early  period,  OT&E  personnel  participate  to 
gain  familiarity  with  the  system  and  to  gain  access 
to  any  test  data  that  can  support  IOT&E.  Next,  the 
combined  portion  of  the  testing  frequently  includes 
shared  objectives  or  joint  data  requirements.  The 
last  segment  normally  contains  the  dedicated  IOT&E 
or  separate  IOT&E  events  to  be  conducted  by  the  OT&E 
agency.  The  OT&E  agency  and  implementing  command 
must  ensure  the  combined  test  is  planned  and  executed 
to  provide  the  necessary  operational  test  information. 

The  OT&E  agency  provides  an  independent  evaluation 
of  the  IOT&E  portion  and  is  ultimately  responsible 
for  achieving  OT&E  objectives. 

The  testing  of  the  Navy's  F- 14  aircraft  has  been  cited  as  an 
example  of  a  successful  combined  T&E  program  (Reference  112).  A  key 
factor  in  the  success  of  the  F- 14  approach  was  the  selection  of  a  T&E 

coordinator  responsible  for  supervising  the  generation  of  test  plans 
that  integrated  the  technical  requirements  of  the  developers  with  the 

operational  requirements  of  the  users.  The  T&E  coordinator  was  also 
responsible  for  the  allocation  of  test  resources  and  the  overall  management 
of  the  test.  In  a  paper  for  the  Defense  Systems  Management  College, 

Mr.  Thomas  Hoivik  describes  the  successful  F- 14  test  program  as  follows: 

The  majority  of  the  Navy  developmental  and  operational 
testing  took  place  during  the  same  period  and  even 

on  the  same  flights.  Maximum  use  was  made  of 
contractor  demonstrations  witnessed  by  the  Navy  testing 
activities  to  obviate  the  retesting  of  a  technical 
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point  already  demonstrated  by  the  contractor. 
Witnessing  by  testing  activities  was  crucially 
important  and  allowed  the  contractor's  data  to  be 
readily  accepted  by  the  testing  activities.  This 
approach  also  helped  to  eliminate  redundancy  in 
testing,  i.e.  the  testing  of  the  same  performance 
parameter  by  several  different  activities  which  has 
been  a  consistent  and  wasteful  feature  Navy  testing 
in  the  past. 

Obviously,  this  approach  placed  a  great  deal  of 
responsibil ity  directly  on  the  shoulders  of  the 
T&E  Coordinator,  and  required  his  staff  to  deal 
knowledgeably  with  a  wide-ranging  and  complex  test 
plan. 


17.3  CONCURRENT  TESTING 

In  1983,  a  senior  DoD  test  and  evaluation  official  testified 
that  a  concurrent  testing  approach  is  usually  not  an  effective  strategy 
(Reference  106).  He  acknowledged,  however,  that  certain  test  events 
may  provide  information  useful  to  both  development  and  operational  testers 
and  that  test  planners  must  be  alert  to  identify  those  events.  His 
testimony  included  the  following  examples  of  situations  where  a  concurrent 
testing  approach  was  unsuccessful: 

(1)  During  AAH  (Advanced  Attack  Helicopter)  testing 
in  1981,  the  Target  Acquisition  Designation  System 
(TADS)  was  undergoing  developmental  and  operational 
testing  at  the  same  time.  The  schedule  did  not  allow 
enough  time  for  qualification  testing  (a  development 
test  activity)  of  the  TADS  prototype  prior  to  a  full 
field  test  of  the  total  aircraft  system,  nor  was 
there  time  to  introduce  changes  to  TADS  problems 
discovered  in  tests.  As  a  result,  the  TADS  performed 
poorly  and  was  unreliable  during  the  operational 
test.  The  resulting  DSARC  action  required  the  Army 
to  fix  and  retest  the  TADS  prior  to  release  of  second 
year  and  subsequent  production  funds. 

(2)  When  the  AIM-7  Sparrow  air-to-air  missile  was 
tested  an  attempt  was  made  to  move  into  operational 
testing  while  developmental  reliability  testing  was 
still  underway.  The  operational  test  was  suspended 
after  less  than  two  weeks  because  of  poor  reliability 
of  the  test  missiles.  The  program  concentrated  on 
an  intensive  reliability  improvement  effort.  A  year 
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after  the  initial  false  start,  a  full  operational 
test  was  conducted  and  completed  successfully. 

(3)  The  Maverick  missile  had  a  similar  experience 
of  being  tested  in  an  operational  environment  before 
component  reliability  testing  was  completed.  As 
a  result,  reliability  failures  had  a  major  impact 
on  the  operational  testers  and  resulted  in  the  program 
being  extended. 

17.4  ADVANTAGES  AND  LIMITATIONS 

Before  adopting  a  combined  or  concurrent  testing  approach, 
program  and  test  managers  are  advised  to  consider  the  advantages  and 
disadvantages  summarized  in  Table  17-1. 

17.5  SUMMARY 

A  combined  or  concurrent  testing  approach  may  offer  an  effective 
means  for  shortening  the  time  required  for  testing  and  also  for  achieving 
cost  savings.  If  such  an  approach  is  used,  extensive  coordination  is 
required  to  ensure  both  the  development  and  operational  requirements 
are  addressed. 

It  is  possible  to  have  combined  test  teams,  consisting  of  both 
DT&E  and  OT&E  personnel,  involved  throughout  the  testing  process.  The 
teams  can  provide  mutual  support  and  share  mutually  beneficial  data, 
as  long  as  the  test  program  is  carefully  planned,  evaluated  and  reporting 
activities  are  conducted  separately. 
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Table  17-1. 


Combined  and  Concurrent  Testing: 
Advantages  and  Limitations 


Conibin  e.d_le^tina 


Advantages 


Limitations 


Shortens  time  required  for  testing  and 
thus,  the  acquisition  cycle. 

Achieves  cost  savings  by  eliminating 
redundant  activities. 

Early  involvement  of  OTSE  personnel 
during  system  development  increases 
their  familiarity  with  system. 

Early  involvement  of  OTSE  personnel 
permits  communication  of  operational 
concerns  to  developer  in  time  to  allow 
changes  in  system  design. 


Requires  extensive  early  coordination. 

Test  objectives  may  be  compromised. 

Requires  development  of  DT/OT  common 
test  data  base. 


If  system  design  is  unstable  and 
far  reaching  modifications  are 
made,  OTSE  must  be  repeated. 

Combined  testing  programs  are  often 
conducted  in  a  development  environment. 

Test  will  be  difficult  to  design  to 
meet  both  DT  S  OT  requirements. 


*  Contractor  personnel  frequently 
perform  maintenance  functions  in  a 
DTSE.  The  system  contractor  i3  prohib¬ 
ited  by  law  from  participating  in  OTSE. 

•  Time  constraints  may  result  in 
less  coverage  than  planned  for 
OTSE  objectives. 


Ccnc.uxx£DiuJI.eaiina 


Advantages 


Limitations 


Shortens  time  required  for  testing  and 
thus,  the  acquisition  cycle. 

Achieves  cost  savings  by  eliminating 
redundant  activities. 


Requires  extensive  coordination. 

If  system  design  is  unstable  and 
far  reaching  modifications  are 
made,  OTSE  must  be  repeated. 


Involvement  of  OTSE  personnel 
permits  communication  of  operational 
concerns  to  developer  in  time  to  allow 
changes  in  system  design. 


Concurrent  testing  programs  are  often 
conducted  in  a  development  environment , 

Contractor  personnel  frequently 
perform  maintenance  functions 
in  a  DTSE.  The  system  contractor 
is  prohibited  by  law  from  partic¬ 
ipating  in  OTSE. 


•  Time  constraints  may  result  in 
less  coverage  than  planned  for 
OTSE  objectives. 


CHAPTER  18 


TEST  RESOURCES 


18.1  INTRODUCTION 

This  chapter  describes  the  various  types  of  resources  available 
for  testing,  explains  test  resource  planning  in  the  Services,  and  discusses 
the  ways  in  which  test  resources  are  funded. 

According  to  DoD  5000.3-M-l,  the  term  "test  resources1'  is  a 
"collective  term  that  encompasses  all  elements  necessary  to  plan,  conduct, 
collect  and  analyze  data  from  a  test  event  or  program."  these  elements 
include  funding  (to  develop  new  or  use  existing  resources),  manpower 
for  test  conduct  and  support,  test  articles,  models,  simulations,  threat 
simulators,  surrogates,  replicas,  test  beds,  special  instrumentation, 
targets,  tracking  and  data  acquisition  instrumentation,  and  equipment 
for  data  reduction,  communications,  meteorology,  utilities,  photography, 
calibration,  security,  recovery,  maintenance  and  repair,  frequency 
management  and  control,  and  base/facility  support  services. 

Existing  ;est  resources  are  in  great  demand  by  competing 
acquisition  programs.  Often  special,  unique  or  one-of-a-kind  test 
resources  must  be  developed  specifically  for  the  test  program.  It  is 
imperative  that  the  requirements  for  these  test  resources  be  identified 
early  in  the  acquisition  cycle  so  that  adequate  funding  can  be  allotted 
for  their  development  and  their  use  can  be  scheduled. 

18.2  OBTAINING  TEST  RESOURCES 

18.2.1  Major  Range  and  Test  Facility  Base 

All  of  the  Services  operate  ranges  and  test  facilities  for 
test,  evaluation,  and  training  purposes.  Twenty-one  of  these  activities 
constitute  the  DoD  Major  Range  and  Test  Facility  Base  (MRTFB),  which 
is  "a  national  asset  which  shall  be  sized,  operated,  and  maintained 
primarily  for  DoD  test  and  evaluation  support  missions,  but  also  is 
available  to  all  users  having  a  valid  requirement  for  its  capabilities. 
The  MRTFB  consists  of  a  broad  base  of  T&E  activities  managed  and  operated 
under  uniform  guidelines  to  provide  T&E  support  to  DoD  Components 
responsible  for  developing  or  operating  materiel  and  weapon  systems." 
(Reference  21).  The  list  of  MRTFB  activities  and  their  locations  are 
shown  in  Figures  18-1  and  18-2.  Summaries  of  the  capabilities  of  each 
of  these  activities  (with  points  of  contact  listed  for  further  information) 
may  be  found  in  DoD  3200. 11-D. 
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MAJOR  RANGE  AND  TEST  FACILITV  BASE  SUMMARV 


White  Sands  Missile  Range 

Kuiajalein  Missile  Range 

Yuma  Prouing  Ground 

Dugway  Prouing  Ground 

Electronic  Prouing  Ground 

Aberdeen  Prouing  Ground 

Pacific  Missile  Test  Center 

Naual  Air  Test  Center 

Naual  Weapons  Center 

Naual  Air  Propulsion  Center 

Atlantic  Undersea  Test  and  Eualuation  Center 

Atlantic  Fleet  Weapons  Training  Facility 

Eastern  Space  and  Missile  Center 

Western  Space  and  Missile  Center 

Arnold  Engineering  Deuelopment  Center 

Tactical  Fighter  Weapons  Center 

Utah  Test  and  Training  Range 

Armament  Diuision  -  3246th  Test  Wing 

Armament  Diuision  -  6585th  Test  Group 

Aeronautical  Systems  Diuision  -  4950th  Test  Wing 


Figure 
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DoD  MRTFB 

Source : 

DoD 

3200. 11-D 
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AIR  FORCE 


DOD  MAJOR  RANGE  AND  TEST  FACILITY  BASE 
LOCATION  OF  ACTIVITIES 


The  MRTFB  facilities  are  available  for  use  by  all  the  Services, 
other  U.S.  Government  agencies,  and,  in  certain  cases,  allied  foreign 
governments  and  private  organizations.  Scheduling  is  based  on  a  priority 
system  and  costs  for  usage  are  billed  uniformly,  as  stated  in  DoD  3200.11. 
The  Deputy  Director,  Defense  Research  and  Engineering  (T&E)  sets  policy 
for  the  composition,  use,  and  test  program  assignments  of  the  MRTFB. 
The  individual  Services  must,  in  turn,  fund,  manage,  and  operate  their 
activities.  They  are  reimbursed  by  each  user  of  the  activity. 

DoD  components  wisning  to  use  an  MRTFB  activity  must  provide 
timely  and  complete  notification  of  their  requirements,  such  as  special 
instrumentation  or  ground  support  equipment  requirements,  to  the  particular 
activity  using  the  documentation  formats  prescribed  by  that  activity. 
The  requirements  must  be  stated  in  the  TEMP  as  will  be  discussed  below. 
Personnel  at  the  MRTFB  activity  will  coordinate  with  prospective  users 
to  assist  them  in  their  T&E  planning,  to  include  conducting  trade-off 
analyses  and  test  scenario  optimization  based  on  test  objectives  and 
test  support  capabilities. 

18.2.2  Service  Test  Facilities 

There  are  other  test  resources  available  in  addition  to  the 
MRTFB.  The  tester  can  determine  current  resources  available  within  the 
Army  by  consulting  documents  such  as  the  Army  Test  Facilities  Register, 
the  Operational  Test  and  Evaluation  Agency  (OTEA),  the  Operational  Test 
Instrumentation  Guide,  and  other  Army  test  agency  and  range  documents. 
Information  on  specific  Navy  test  resources  is  found  in  the  users  manuals 
published  by  each  range  and  the  Commander  Operational  Test  and  Evaluation 
Force  (COMOPTEVFOR)  catalog  of  available  support. 

18.3  TEST  RESOURCE  PLANNING 

The  development  of  special  test  resources  to  support  a  weapon 
system  test  can  be  costly  and  time  consuming.  This,  coupled  with  the 
high  demand  for  existing  test  resources  and  facilities,  requires  that 
early  planning  be  accomplished  to  determine  all  test  resource  requirements 
for  weapon  system  T&E.  The  tester  must  plan  and  conduct  tests  to  take 
full  advantage  of  the  existing  investment  in  the  DoD  ranges,  facilities 
and  test  resources  (collectively,  the  MRTFB). 

Problems  associated  with  range  and  facility  planning  are  that 
priority  systems  tend  to  get  top  priority,  i.e.,  B-1B,  M-l,  etc.  Range 
schedules  are  often  in  conflict  due  to  system  problems  which  occur  during 
testing,  and  there  is  often  a  shortage  of  funds  to  complete  testing. 

18.3.1  TEMP  Requirements 

The  Program  Manager  must  state  all  test  resource  requirements 
in  the  TEMP,  and  must  include  items  such  as  unique  instrumentation,  threat 
simulators,  surrogates,  targets,  and  test  articles.  Included  in  the 
TEMP  are  a  critical  analysis  of  anticipated  resource  shortfalls,  their 
effect  on  system  T&E,  and  plans  to  correct  resource  deficiencies.  The 
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initial  TEMP  must  be  prepared  before  Milestone  I,  hence  initial  test 
resource  planning  must  be  accomplished  during  the  concept  exploration/ 
definition  phase.  Refinements  and  reassessments  of  test  resource 
requirements  are  included  in  each  annual  and  milestone  TEMP  update. 
The  required  content  of  the  test  resource  summary  section  of  the  TEMP 
is  in  Part  V  -  Test  and  Evaluation  Resource  Summary,  DoD  5000.3-M-l. 


18.3.2  Service  Test  Resource  Planning 

More  detailed  listings  of  required  test  resources  are  generated 
in  conjunction  with  the  detailed  test  plans  written  by  the  materiel 

developer  and  operational  tester.  These  test  plans  describe  test 
objectives,  measures  of  effectiveness  (MOE),  scenarios,  and  specific 

test  resource  requirements. 

18.3.2.1  Army  Test  Resource  Planning 

In  the  Army,  the  operational  tester  prepares  the  Test, 

Evaluation,  Analysis,  and  Modeling  (TEAM)  plan  and  the  Independent 
Evaluation  Plan  (IEP),  which  are  the  primary  planning  documents  for  OT&E 
of  the  weapon  system.  These  documents,  which  should  be  prepared  early 

in  the  acquisition  cycle  (at  the  beginning  of  the  Concept  Demonstration 

and  Validation  Phase),  describe  the  entire  T&E  approach,  including  critical 
issues,  test  methodology,  measures  of  effectiveness,  and  all  necessary 

test  resources.  The  TEAM  Plan  and  IEP  provide  the  primary  inputs  to 
the  Outline  Test  Plan,  which  contains  a  detailed  description  of  each 
identified  required  test  resource,  where  and  when  it  is  to  be  provided, 
and  the  organization,  usually  the  Forces  Command  (FORSCOM)  or  Training 
and  Doctrine  Command  (TRADOC),  that  will  provide  the  resource. 

The  tester  must  coordinate  the  Outline  Test  Plan  (OTP)  with 

all  major  commands  or  agencies  expected  to  provide  test  resources.  Then, 
the  OTP  is  submitted  to  the  Resource  Management  Division,  HQ,  OTEA,  for 
review  by  the  Test  Schedule  and  Review  Committee  (TSARC)  and  incorporation 
into  the  Army's  Five  Year  Test  Program  (FYTP).  The  initial  OTP  for  each 
test  should  be  submitted  to  the  TSARC  five  years  prior  to  the  test's 
start  date.  Revised  OTPs  are  submitted  as  more  information  becomes 
available  or  requirements  change,  but  a  final  comprehensive  version  of 

the  OTP  should  be  submitted  at  least  a  year  before  the  resources  are 
required. 


The  TSARC  is  responsible  for  providing  high-level  centralized 
management  of  OT&E  resource  planning.  The  TSARC  is  chaired  by  the 
Commanding  General  OTEA,  and  consists  of  general  officer  or  equivalent 
representatives  from  the  Army  staff  and  major  commands.  The  TSARC  meets 
semiannually  to  review  all  OTPs,  resolve  conflicts,  and  coordinate  all 
identified  test  resource  requirements  for  inclusion  in  the  FYTP.  The 
FYTP  is  a  formal  resource  tasking  document  for  current  and  near  term 
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tests  and  a  planning  document  for  tests  scheduled  for  the  out-years. 
All  OTPs  are  reviewed  during  the  semiannual  reviews  to  ensure  that  any 
refinenents  or  revisions  are  approved  by  the  TSARC  and  reflected  in  the 
FYTP.  The  FYTP  is  produced  as  a  hard  copy  document  by  OTEA  and  is  also 
maintained  in  automated  format  on  the  TRADOC  Resource  Management  System 
(TRMS).  This  system  is  maintained  at  Ft.  Hood,  Texas,  with  access 
terminals  located  at  Army  major  headquarters  and  test  boards. 

The  TSARC-approved  OTP  is  a  tasking  document  by  which  the  tester 
requests  Army  test  resources.  The  TSARC  coordinates  resource  requests, 
sets  priorities,  resolves  conflicts,  and  schedules  resources.  The 
resultant  FYTP,  when  approved  by  the  DCSOPS,  HQ  DA,  is  a  formal  tasking 
document  that  reflects  the  agreements  made  by  the  resource  providers 
(AMC,  TRADOC,  FORSCOM,  etc.)  to  make  the  required  test  resources  available 
to  the  designated  tests.  If  test  resources  from  another  Service,  a  non-DoD 
governmental  agency  (such  as  DOE  or  NASA),  or  a  contractor  are  required, 
the  request  is  coordinated  by  the  OTEA  Resource  Management  Division. 
For  example,  the  request  for  a  range  must  be  made  at  least  2  years  in 
advance  to  ensure  availability.  However,  due  to  the  long  lead  time 
required  to  schedule  these  non-Army  resources,  their  availability  cannot 
be  guaranteed  if  the  test  is  delayed  or  retesting  is  required.  The  use 
of  resources  outside  the  U.S.,  such  as  in  Canada,  Germany,  or  other  NATO 
countries  is  also  handled  by  OTEA. 

18.3.2.2  Navy  Test  Resource  Planning 

In  the  Navy,  the  developing  agency  and  the  operational  tester 
are  responsible  for  identifying  the  specific  test  resources  which  are 
required  in  testing  the  weapon  system.  In  developing  the  requirements 
for  test  resources,  the  Program  Manager  (PM)  and  Operational  Test  Director 
(OTD)  refer  to  documents  such  as  the  MNS,  SCP,  DCP,  Navy  Decision 
Coordinating  Paper  (NDCP),  Operational  Requirement  (OR),  threat 
assessments,  OPNAVINST  3960. IOC  (Test  and  Evaluation),  and  the  OTD  Guide 
(COMOPTEVFOR  Instruction  3960. ID).  Upon  Chief  of  Naval  Operations 
approval,  the  TEMP  becomes  the  controlling  management  document  for  all 
T&E  of  the  weapon  system;  it  constitutes  CNO  direction  to  conduct  the 
T&E  program  defined  in  the  TEMP,  including  the  commitment  of  Research, 
Development,  Test  and  Evaluation  financial  support,  and  it  commits  fleet 
units  and  schedules.  It  is  prepared  by  the  Program  Manager,  with  OT&E 
inputs  provided  by  the  COMOPTEVFOR  Operational  Test  Director.  The  TEMP 
defines  all  T&E  (DT&E,  OT&E,  and  PAT&E)  to  be  conducted  for  the  system 
and  describes  the  test  resources  required  in  as  much  detail  as  possible. 

The  Navy  utilizes  its  operational  naval  forces  to  provide 
realistic  T&E  of  new  weapons  systems.  Each  year,  the  CNO  (OP-098)  compiles 
all  fleet  support  requirements  for  RDT&E  program  support  from  the  TEMPs 
and  publishes  the  CNO  Long  Range  RDT&E  Support  Requirements  document 
for  the  budget  and  out-years.  In  addition,  a  quarterly  forecast  of  support 
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requirements  is  published  approximately  5  months  before  the  Fleet 
Employment  Scheduling  Conference  for  the  Quarter  in  which  the  support 
is  required.  These  documents  summarize  OT&E  requirements  for  Fleet 
services  and  are  used  by  the  Fleet  for  scheduling  services  and  out-year 
budget  projections. 

Requests  for  use  of  range  assets  are  usually  initiated  informally 
with  a  phone  call  from  the  PM  and/or  OTD  to  the  range  manager,  followed 
by  formal  documentation.  Requests  for  Fleet  support  are  usually  more 
formal.  COMOPTEVFOR,  in  coordination  with  the  PM,  forwards  the  TEMP 
and  a  Fleet  RDT&E  Support  Request  to  the  CNO.  Upon  approval  of  the 
request,  the  CNO  tasks  the  Fleet  CINC  by  letter  or  message  to  coordinate 
with  OPTEVFOR  to  provide  the  requested  support. 

Use  of  most  Navy  ranges  must  be  scheduled  at  least  a  year  in 
advance.  Each  range  consolidates  and  prioritizes  user  requests,  negotiates 
conflicts,  and  attempts  to  schedule  range  services  to  satisfy  all  requests. 
If  the  desired  range  services  cannot  be  made  available  when  required, 
the  test  must  wait,  or  the  CNO  resolves  the  conflict.  Because  ranges 
are  fully  scheduled  in  advance,  it  is  difficult  to  accommodate  a  test 
that  is  delayed  or  requires  additional  range  time  beyond  that  originally 
scheduled.  Again,  the  CNO  can  examine  the  effects  of  delays  or  retest 
requirements  and  issue  revised  priorities,  as  required. 

Requests  for  use  of  non-Navy  OT&E  resources  are  initiated  by 
COMOPTEVFOR.  OPTEVFOR  is  authorized  direct  liaison  with  the  other  Service 
independent  operational  test  agencies  (OTA)  to  obtain  OTA-controlled 
resources.  Requests  for  other  government-owned  resources  are  forwarded 
to  the  CNO  (OP-098)  for  formal  submission  to  the  Service  Chief  (for  Service 
assets)  or  to  the  appropriate  government  agency  (e.g.,  DOE  or  NASA). 
Use  of  contractor  resources  is  usually  handled  by  the  PM,  although 
contractor  assets  are  seldom  required  in  OT&E,  since  the  Fleet  is  used 
to  provide  an  operational  environment.  Requests  for  use  of  foreign  ranges 
are  handled  by  the  OP-098  Assistant  for  International  R&D  (0P-098F). 

18.3.2.3  Air  Force  Test  Resource  Planning 

The  test  resources  required  for  test  and  evaluation  (T&E)of 
an  Air  Force  weapon  system  are  identified  in  detail  in  the  Test  Program 
Outline  (TPO),  which  is  prepared  by  the  responsible  Air  Force  T&E 
organization.  In  general,  AFOTEC  is  that  test  organization  for  major 
programs  (those  requiring  Defense  Acquisition  Board  (DAB)  or  AFSARC 
review),  while  a  Service  Major  Command  test  agency  would  conduct  the 
test  for  non-major  programs,  with  AFOTEC  monitoring  and  providing 
assistance,  as  required. 

During  the  advanced  planning  phase  of  a  weapon  system  acquisition 
(5-6  years  prior  to  OT&E),  AFOTEC  (TE/XP)  prepares  the  OT&E  section  of 
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the  first  full  TPO,  coordinates  the  TPO  with  all  supporting  organizations, 
and  assists  the  resource  manager  (AFOTEC/RM)  in  programming  of  required 
resources.  The  resource  requirements  listed  in  the  TPO  are  developed 
by  the  resource  manager,  test  manager,  and  Operational  Test  Director 
using  sources  such  as  the  statement  of  operational  need  (SON)  and  threat 
assessments.  The  TPO  should  specify  in  detail  all  the  resources  necessary 
to  successfully  conduct  a  test. 

The  TPO  is  the  formal  means  by  which  test  resource  requirements 
are  communicated  to  the  Air  Staff  and  to  the  appropriate  commands  and 
agencies  tasked  to  supply  the  needed  resources.  Hence,  if  a  required 
resource  is  not  specified  in  the  TPO,  it  is  likely  that  the  resource 
will  not  be  available  for  the  test.  The  TPO  is  revised  and  updated  every 
six  months  ,  since  the  test  resource  requirements  become  better  defined 
as  the  OT&E  plans  mature.  The  initial  TPO  serves  as  a  baseline  for 
comparison  of  planned  OT&E  resources  with  actual  expenditures.  Comparisons 
of  the  initial  TPO  with  subsequent  updates  provides  an  audit  trail  of 
changes  in  the  test  program  and  its  testing  requirements.  AFOTEC  maintains 
all  TPOs  on  a  computer  data  base,  which  permits  immediate  response  to 
all  queries  regarding  test  resource  requirements. 

AFOTEC/RM  consolidates  the  resource  requirements  from  all  TPOs, 
both  AFOTEC  and  MAJCOM  prepared,  with  participating  and  supporting 
organizations  and  agencies  outside  AFOTEC.  Twice  yearly,  the  RM  office 
prepares  a  draft  of  the  USAF  Program  for  OT&E  (the  Program  Outline  ( PO ) ) , 
which  is  a  master  planning  and  programming  document  for  resource 
requirements  for  all  HQ  USAF-directed  OT&E,  and  distributes  it  to  all 
concerned  commands,  agencies,  and  organizations  for  review  and 
coordination.  The  PO  is  then  submitted  to  the  Air  Staff  for  review  and 
approval  by  the  Operational  Resource  Management  Assessment  System  for 
Test  and  Evaluation  (ORMAS/TE),  which  operates  under  the  authority  of 
HQ  AF/XOORE.  The  ORMAS  Board  is  composed  of  HQ  USAF  action  officers 
and  senior  officers  from  MAJCOMs  and  agencies  involved  in  OT&E;  it  meets 
semiannually  to  review  the  PO,  identify  the  impacts  of  OT&E  resource 
requests  on  operational  command  mission  requirements,  and  resolve  such 
impacts  and  conflicting  requirements  at  the  appropriate  Air  Staff  level. 
Through  the  ORMAS  process,  HQ  USAF  approves  the  PO,  and  it  becomes 
directive  to  all  participants  to  take  planning,  programming,  and  budgeting 
actions.  Agreements  made  among  ORMAS  participants  regarding  TPO  and 
PO  resource  requirements  are  considered  binding. 

All  requests  for  test  resources  are  coordinated  by  HQ  AFOTEC 
as  part  of  the  TPO  preparation  process.  When  a  new  weapon  system 
development  is  first  identified,  AFOTEC  provides  a  Test  Manager  (TM) 
who  begins  long  term  OT&E  planning.  The  TM  begins  identifying  needed 
test  resources,  such  as  instrumentation,  simulators,  and  models,  and 
works  with  the  AFOTEC/RM  directorate  to  obtain  them.  If  the  required 
resource  does  not  belong  to  AFOTEC,  AFOTEC/RM  will  negotiate  with  the 
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commands  having  the  resource.  In  the  case  of  models  and  simulators, 
AFOTEC  surveys  what  is  available,  assesses  credibility,  then  coordinates 
with  the  owner  or  developer  to  use  it.  The  Joint  Technical  Coordinating 
Group  publishes  a  document  on  EW  models. 

Range  scheduling  should  be  done  as  early  as  possible.  At  least 
a  year  is  required,  but  often  a  test  can  be  accommodated  with  a  few  months 
notice  if  there  is  no  requirement  for  special  equipment  or  modifications 
to  be  provided  at  the  range.  Some  of  the  Air  Force  ranges  are  scheduled 
well  in  advance  and  cannot  accommodate  tests  which  encounter  delays  or 
retest  requirements. 

AFOTEC/RM  attempts  to  resolve  conflicts  among  various  systems 
competing  for  scarce  test  resources  and  can  elevate  the  request  to  the 
Commander,  AFOTEC  if  necessary.  Decisions  on  resource  utilization  and 
scheduling  are  based  on  the  weapon  system’s  assigned  priority. 

AFOTEC/RM  and  the  Test  Manager  also  arrange  for  use  of  the 
resources  of  other  Services,  non-DoD  government  agencies,  and  contractors. 
Use  of  non-U. S.  resources,  such  as  a  Canadian  range,  are  coordinated 
by  AF/XOORE  and  based  on  formal  Memoranda  of  Understanding  (MOU). 
USAFE/DOQ  handles  requests  for  European  ranges.  Use  of  a  contractor-owned 
resource,  such  as  a  model,  is  often  obtained  through  the  System  Program 
Office  (SPO)  or  a  General  Support  Contract. 

18.4  TEST  RESOURCE  FUNDING 

The  Five  Year  Defense  Program  (FYDP)  is  the  basic  DoD  programming 
document  that  records,  summarizes,  and  displays  Secretary  of  Defense 
(SECDEF)  decisions.  In  the  FYDP,  costs  are  divided  into  three  categories 
for  each  program  element:  research  and  development  costs,  investment 
costs,  and  operating  costs.  Congress  appropriates  to  the  Office  of 
Management  and  Budget  (0MB)  who  apportions  funding  through  the  SECDEF 
to  the  Services  and  to  other  Defense  Agencies  who  allocate  funds  to  others 
(claimants,  sub-claimants,  administering  offices,  commanding  generals, 
etc. ). 

The  Planning,  Programming,  and  Budgeting  System  (PPBS)  is  a 
DoD  internal  system  that  is  used  to  develop  the  inputs  to  Congress  for 
each  year's  budget  while  developing  future  year  budgets.  The  PPBS  is 
milestone  oriented.  There  are  four  concurrent  PPBS  cycles  ongoing  at 
one  time.  These  cycles  are:  planning,  programming,  budgeting,  and 
execution/enactment.  At  any  one  time  there  are  three  budgets  being  worked 
by  the  Services.  The  current  fiscal  year  budget  is  being  executed,  the 
following  year's  budget  is  being  programmed,  and  the  next  year's  budget 
is  being  planned. 
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There  are  six  types  of  funding  in  the  PPBS:  Research  funding 
for  maintaining  the  technology  base;  Exploratory  Development  funding 
for  conducting  the  Concept  Exploration/Definition  phase;  Advanced 
Development  funding  for  conducting  both  the  Concept  Exploration/Definition 
phase  and  the  Demonstration/Validation  phase;  Engineering  Development 
funding  for  conducting  the  Full-Scale  Development  phase;  Operational 
Systems  Development  funding  for  conducting  the  Production  and  Deployment 
phase;  and  Research,  Development,  Test  and  Evaluation  (RDT&E)  management 
and  support  funding  which  is  used  throughout  the  development  and  production 
cycle  until  the  system  is  operationally  deployed,  where  Operations  and 
Maintenance  ( O&M)  funding  is  used.  The  RDT&E  appropriation  funds  the 
costs  associated  with  research  and  development,  including  test  items, 
DT&E  and  test  support  of  OT&E  of  the  system  or  equipment  and  the  test 
i terns . 


The  funding  that  is  planned,  programmed,  and  budgeted  through 
the  PPBS  cycle  is  not  always  the  same  funding  amount  that  Congress 
appropriates  or  the  Program  Manager  (PM)  receives.  If  the  required  funding 
for  a  test  program  is  not  authorized  by  Congress,  the  Program  Manager 
has  four  choices  of  how  to  react.  The  PM  can  submit  a  supplemental  budget 
(for  unfunded  portions  of  his  program),  request  deficiency  funding  (for 
unforeseen  program  problems),  use  transfer  authority  (from  other  programs 
within  his  Service),  or  he  can  reprogram  the  funds  (to  restructure  his 
program) . 


Generally,  testing  accomplished  for  a  specific  system  prior 
to  the  production  decision  is  funded  from  Research,  Development,  Test 
and  Evaluation  (RDT&E)  appropriations,  while  testing  accomplished  after 
the  production  decision  is  funded  from  other  procurements,  operations 
and  maintenance  (O&M)  appropriations.  Testing  of  product  improvements, 
block  upgrades,  and  major  modifications  is  funded  from  the  same 
appropriations  as  the  program  development.  Follow-on  Test  and  Evaluations 
(FOT&E)  are  normally  funded  from  O&M  funds. 

Funding  associated  with  T&E  (including  instrumentation,  targets 
and  simulations)  are  identified  in  the  system  acquisition  cost  estimates. 
Service  acquisition  plans,  and  the  TEMP.  General  funding  information 
for  development  and  operational  tests  are  as  follows: 

Development  Test  Funding.  Funds  required  to  conduct  engineering 
and  development  tests  are  programmed  and  budgeted  by  the  material 
developer,  based  upon  the  requirements  of  the  TEMP.  These  costs  may 
include,  but  are  not  limited  to,  procuring  test  sampler/prototypes,  support 
equipment,  transportation  costs,  technical  data,  training  of  test 
personnel,  repair  parts,  and  test  specific  instrumentation,  equipment 
and  facilities. 
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Operational  Test  Funding.  Funds  required  to  conduct  OT  are 
programmed  and  budgeted  by  the  Service  operational  test  agency  or 
organization.  The  funds  are  programmed  in  the  Service's  long  range  test 
program,  and  the  funds  requirements  are  obtained  from  the  test  resourcing 
documentation  and  TEMP. 

The  Program  Manager  must  pay  for  the  use  of  all  test  resources, 
such  as  the  MRTFB,  and  for  the  development  of  specialized  resources  needed 
specifically  for  testing  the  weapon  system  being  developed. 

18.4.1  Army  Funding 

Test  resources  are  developed  and  funded  under  various  Army 
appropriations.  The  Army  Materiel  Com.  and  and  its  commodity  commands 
provide  test  items,  spare  parts,  support  items  such  as  diagnostic 
equipment,  and  ammunition.  Soldiers,  ranges,  fuel,  test  support  personnel, 
and  maneuver  areas  are  provided  by  TRADOC  or  FORSCOM.  The  weapon  system 
program  manager  uses  RDT&E  funds  to  reimburse  these  supporting  commands 
for  costs  directly  related  to  his  test.  The  weapon  system  material 
developer  is  also  responsible  for  funding  the  development  of  new  test 
resources  specifically  needed  to  test  the  weapon  system.  Examples  of 
such  special  purpose  resources  include  models,  simulations,  special 
instrumentation  and  test  equipment,  range  modifications,  EW  simulators, 
and  sometimes  threat  simulators.  Although  the  Army  has  a  separate  budget 
and  development  plan  for  threat  simulators,  the  Army  Development  and 
Acquisition  of  Threat  Simulators  (ADATS)  program,  many  weapon  system 
developers  still  have  to  fund  the  cost  of  new  threat  systems  that  are 
specifically  needed  to  test  their  weapon  system. 

18.4.2  Navy  Funding 

In  the  Navy,  the  weapon  system  PM  is  responsible  for  funding 
the  development  of  all  required  test-specifir  resources,  such  as  test 
articles,  expendables,  one-of-a-kind  targets,  ta  collection/reduction, 
and  instrumentation,  out  of  the  program's  RDT&E  funds.  The  development 
of  generic  test  resources,  such  as  targets,  threat  simulators,  and  range 
capabilities,  that  can  be  used  in  OT&E  of  multiple  weapons  systems,  is 
funded  from  OPNAV  generic  accounts  (such  as  target  development)  and  not 
from  weapon  system  RDT&E.  The  PM's  RDT&E  funds  pay  for  all  DT  and  OT 
through  OPEVAL.  The  PM  pays  for  all  post  production  OT  with  program 
funds. 

18.4.3  Air  Force  Funding 

In  the  Air  Force,  direct  cost  funding  requires  that  test-peculiar 
(direct)  costs  associated  with  a  particular  test  program  be  reimbursed 
by  the  System  Program  Office  to  the  designated  test  agency.  The  RDT&E 


appropriation  funds  the  cost  associated  with  research  and  development, 
including  test  items,  DT&E  and  Air  Force  Systems  Command  (AFSC)  support 
of  OT&E  of  the  system  or  equipment  and  the  tests  items.  Costs  associated 
with  IOT&E  are  RDT&E  funded  and  costs  of  QOT&E  are  Production  funded. 
The  IOT&E  Manager  prepares  a  Test  Plan  Outline  (TPO)  that  summarizes 
the  resource  requirements  for  IOT&E  and  related  test  support.  All  pretest 
IOT&E  planning  is  budgeted  through  and  paid  out  of  the  operation  account 
of  AFOTEC  or  MAJCOM  responsible  for  operating  the  system  and  funded  by 
the  O&M  appropriation.  The  actual  IOT&E  is  budgeted  through  and  paid 
out  of  the  RDT&E  appropriation  and  directed  to  AFOTEC  by  HQ  AFSC.  FOT&E 
costs  are  paid  by  AFOTEC  and/or  the  MAJCOM  operating  the  system  and  funded 
by  the  O&M  appropriation. 

18.5  SUMMARY 

Test  resources  are  in  great  demand  and  their  use  must  be 
scheduled  well  in  advance  of  a  test.  Resources  specific  to  a  particular 
test  must  often  be  developed  and  funded  from  the  PM's  own  RDT&E  budget. 
Thus,  the  PM  and  his  testers  must  ensure  that  test  resource  requirements 
are  identified  early  in  the  acquisition  cycle,  that  they  are  documented 
in  the  initial  TEMP,  and  that  modifications  and  refinements  are  reported 
in  the  annual  TEMP  updates. 

Funds  for  testing  are  provided  by  Congressional  appropriation 
to  the  0MB  who  apportions  the  funds  to  the  to  the  Services  through  the 
SECDEF.  The  PPBS  is  the  DoD  process  used  to  formulate  budget  requests 
to  Congress.  The  Program  Managers  requests  for  test  resources  are  usually 
outlined  in  the  TEMP.  Generally,  System  development  is  funded  from  RDT&E 
funds  until  the  system  is  operationally  deployed  and  maintained.  The 
O&M  funds  are  used  for  FOT&E  and  system  maintenance.  The  weapon  system 
material  developer  is  also  responsible  for  funding  the  development  of 
new  test  resources  specifically  needed  to  test  the  weapon  system. 
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CHAPTER  19 

TESTING  FOR  VULNERABILITY  AND  LETHALITY 


19.1  INTRODUCTION 

This  chapter  addresses  the  need  to  explore  the  vulnerability 
and  lethality  aspects  of  a  system  through  test  and  evaluation  practices 
and  procedures.  In  particular,  this  chapter  describes  the  recently 
mandated  Live  Fire  Test  Program  which  has  been  established  to  evaluate 
the  vulnerability  and  lethality  of  developing  systems.  It  also  discusses 
the  role  of  test  and  evaluation  in  assessing  a  system's  ability  to  perform 
in  a  nuclear  combat  environment.  The  discussion  of  testing  for  nuclear 
survivability  is  based  primarily  on  information  contained  in  the  "Nuclear 
Survivability  Handbook  for  OT&E,"  prepared  by  the  Air  Force  Operational 
Test  and  Evaluation  Center  (Reference  91). 

19.2  LIVE  FIRE  TESTING 
19.2.1  Background 

The  National  Defense  Authorization  Act  for  Fiscal  Year  1987 
(Reference  97)  established  a  formal  requirement  for  vulnerability  and 
lethality  testing.  Specifically,  this  law  requires  that: 

(1)  a  major  system  "may  not  proceed  beyond  low-rate 
initial  production  until  realistic  survivability 
testing  of  the  system  is  completed.  .  . "  and 

(2)  "a  major  munitions  program  or  missile  program 
may  not  proceed  beyond  low-rate  initial  production 
until  realistic  lethality  testing  of  the  program 
is  completed.  .  . " 


An  amendment  to  this  law  has  been  proposed  that  will  substitute 
the  word  "vulnerability"  for  "survivability".  This  will  bring  the 
legislation  in  closer  alignment  with  DoD  definitions  of  these  two  terms. 
Survivability  is  a  broad  concept  that  involves  consideration  of  other 
factors  such  as  susceptibility  to  hostile  attack.  Vulnerability  (from 
a  defensive  point  of  view)  or  lethality  (from  an  offensive  point  of  view) 
refer  to  a  measure  of  the  probability  of  a  kill,  given  a  hit  (Reference 
37).  Table  19-1  summarizes  the  relationships  between  the  concepts  of 
survivability,  effectiveness,  vulnerability,  lethality,  and  susceptibility. 

Guidelines  in  the  legislation  further  stipulate  that 
survivability  and  lethality  tests  must  be  conducted  "sufficiently  early 
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in  the  development  phase  of  the  system  or  program  to  allow  any  design 
deficiency  demonstrated  by  the  testing  to  be  corrected  .  .  .  before 
proceeding  beyond  low-rate  initial  production." 

The  legislation  contains  the  following  definitions  of 
survivability  and  lethality  testing: 

(1)  Survivability  testing  means  "testing  for 
vulnerability  and  survivability  of  the  system  in 
combat  by  firing  munitions  likely  to  be  encountered 
in  combat.  .  .at  the  system  configured  for  combat, 
with  the  primary  emphasis  on  testing  vulnerability 
with  respect  to  potential  user  casualties.  .  ." 

(2)  Lethality  testing  means  "testing  for  lethality 
by  firing  the  munition  or  missile  concerned  at 
appropriate  targets  configured  for  combat." 


In  these  definitions  the  term  "configured  for  combat"  is 
understood  to  mean  a  weapon  system,  platform,  or  vehicle  that  is  equipped 
with  all  dangerous  materials,  including  all  flammables  and  explosives, 
that  would  normally  be  on  board  in  combat. 

The  Live  Fire  Test  Program  is  an  outgrowth  of  the  OSD  Joint 

Live  Fire  Test  Program.  The  fundamental  difference  in  the  two  programs 

is  that  the  Live  Fire  Test  Program  has  been  established  to  examine  systems 
that  are  still  in  development  while  the  Joint  Live  Fire  Program  tests 

systems  that  are  already  in  field  use. 

The  Live  Fire  Test  Program  is  administered  by  the  Deputy  Director 
Defense  Research  and  Engineering(Test  and  Evaluation)  (DDDR&E(T&E) ) . 
His  responsibilities  with  regard  to  the  Live  Fire  Test  Program  are  listed 
in  Table  19-2.  Each  of  the  Services  has  identified  a  focal  point  for 

Live  Fire  Testing.  By  September  1987,  discussions  between  OSD  and  the 

Services  had  resulted  in  the  identification  of  a  total  of  44  weapons 
systems  for  live  fire  testing. 

19.2.2  Live  Fire  Tests 

There  are  varying  types  and  degrees  of  live  fire  tests.  The 

matrix  in  Table  19-3  illustrates  the  various  possible  combinations. 
Of  them,  full-scale,  full-up  testing  is  usually  considered  to  be  the 

most  realistic  and  is  the  type  of  testing  called  for  in  the  National 

Defense  Authorization  for  Act  FY87. 

The  importance  of  full-scale  testing  has  been  well  demonstrated 
by  the  recent  Joint  Live  Fire  tests.  In  one  case,  these  tests  contradicted 
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Table  19-2.  Live  Fire  Test  Responsibilities  of  the  DDDRE(T&E) 

•  Implement  all  aspects  of  the  Live  Fire  Test  (LFT)  Program 

•  Develop  and  recommend  LFT  policy 

•  Ensure  Service  LFT  programs  are  realistic  and  relevant 

•  Evaluate  LFT  plans  prepared  by  Services 

•  Direct/participate  in  making  independent  assessments  of 
Service  Live  Fire  Tests  and  Evaluations 

•  Improve  the  conduct  of  LFT  programs 

•  Develop  instrumentation  and  facilities  for  LFT 

•  Acquire  foreign  targets  and  ammunition 


Source:  Statement  by  the  Assistant  Deputy  Under  Secretary 

of  Defense  (Live  Fire  Test)  Before  the  Acquisition 
Policy  Panel  of  the  House  Armed  Services  Committee, 
September  10,  1987. 


Table  19-3.  Types  of  Live  Fire  Testing 
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Scale _ 

Full  Scale 


Sub- scale 


Loading 


Fu  1 1-  up  _  _ _ _ _ 

Complete  System:  with 
combustibles  (e.g.,  Bradley 
Phase  II  tests,  aircraft 
"proof"  tests) 


Components,  subcomponents: 
with  combust ibles  (e.g.,  fuel 
cell  tests  behind  armor  mock-up 
aircraft  engine  fire  tests) 


Inert  a _ _ _ _ _ _ 

Complete  system:  no 
combustibles  (e.g., 
tests  of  new  armor  on 
actual  tanks,  aircraft 
flight  control  tests) 

Components,  subcomponents, 
structures,  terminal  ballistics, 
munitions  performance,  behind- 
armor  tests,  warhead 
characterization  (e.g.,  armor/ 
wa rhead  interaction  tests,  aircraft 
component  structural  tests) 


a 


In  some  cases,  targets  are  "semi- inert" 
(Example:  tests  of  complete  tanks  with 


meaning  some  combustibles 
fuel  and  hydraulic  fluid. 


are  on  board  but  not  all 
but  dummy  ammunition.) 


Source  : 


"Live  Fire  Testing:  Evaluating  DoD ' s  Programs, 

General  Accounting  Office,  GAO/PEMD-87-17,  August  1987 
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earlier  conclusions  concerning  the  flammability  of  a  new  hydraulic  fluid 
used  in  F- 15  and  F- 16  aircraft.  Laboratory  tests  had  demonstrated  that 
the  new  fluid  was  less  flammable  than  the  standard  fluid.  However,  during 
the  JLF  tests,  30  percent  of  the  shots  on  the  new  fluid  resulted  in  fires 
contrasted  with  15  percent  of  the  shots  on  the  standard  fluid  (Reference 
100). 


While  much  insight  and  valuable  wisdom  is  to  be  obtained  through 
the  testing  of  components  or  subsystems,  some  phenomena  are  only  observable 
when  full -up  systems  are  tested.  The  interaction  of  such  phenomena  has 
been  termed  "cascading  damage."  Such  damage  is  a  result  of  the  synergistic 
damage  mechanisms  that  are  at  work  in  the  "real  world"  and  likely  to 
be  found  during  actual  combat.  Live  Fire  Testing  provides  a  means  for 
examining  the  damages  inflicted  not  only  on  materiel  but  also  on  personnel. 
The  crew  casualty  problem  is  an  important  issue  that  the  LFT  program 
is  addressing.  The  program  provides  an  opportunity  to  assess  the  effects 
of  the  complex  environments  that  crews  are  likely  to  encounter  in  combat 
(e.g.,  fire,  toxic  fumes,  blunt  injury  shock,  and  acoustic  injuries) 
(Reference  91). 

19.2.3  Use  of  Modeling  and  Simulation 

Survivability  and  lethality  assessments  have  traditionally 
relied  to  a  great  extent  on  the  use  of  modeling  and  simulation  techniques. 
The  Live  Fire  Test  Program  does  not  replace  the  need  for  such  techniques; 
in  fact,  the  Live  Fire  Test  Guidelines  Issued  by  OSD  in  May  1987  require 
that  no  shots  be  conducted  until  pre-shot  model  predictions  are  made 
concerning  the  expected  damage.  Such  predictions  are  useful  for  several 
reasons.  First,  they  assist  in  the  test  planning  process.  If  a  model 
predicts  that  no  damage  will  be  inflicted,  test  designers  and  planners 
should  re-examine  the  selection  of  the  shotlines  and/or  re-assess  the 
accuracy  of  the  threat  representation.  Second,  pre-shot  model  predictions 
provide  the  Services  with  the  opportunity  to  validate  the  accuracy  of 
the  models  by  comparing  them  with  actual  LFT  results.  At  the  same  time, 
the  LFT  program  reveals  areas  of  damage  that  may  be  totally  absent  from 
existing  models  and  simulations.  Third,  pre-shot  model  predictions  can 
be  used  to  help  conserve  scarce  target  resources.  For  example,  models 
can  be  used  to  determine  a  sequence  of  shots  that  provides  for  the  less 
damaging  shots  to  be  conducted  first,  followed  by  the  more  catastrophic 
shots  resulting  in  maximum  target  damage. 

19.2.4  Live  Fire  Test  Guidelines 

Live  Fire  Test  Guidelines  were  issued  in  May  1987  by  OSD  in 
support  of  DoD  5000.3-M-l,  "Test  and  Evaluation  Master  Plan  (TEMP) 
Guidelines."  Although  not  formally  approved,  the  guidelines  state  that 
plans  for  live  fire  testing  must  now  be  included  in  the  TEMP.  Key  points 
covered  in  the  LFT  Guidelines  include  the  following: 
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o  The  Live  Fire  Test  and  Evaluation  (LFT&E)  plan  is 

the  basic  planning  document  used  by  OSD  and  the  Services 
to  plan,  review,  and  approve  LFT&E. 

o  The  LFT&E  plan  must  contain  general  information  on 
the  system's  required  performance,  operational  and 
technical  characteristics,  critical  test  objectives, 
and  the  evaluation  process. 

o  Each  LFT&E  plan  must  include  testing  of  complete 

systems.  A  limited  set  of  live  fire  tests  may  involve 
production  components  configured  as  a  subsystem  prior 
to  full -up  testing. 

o  A  Service  report  must  be  submitted  within  30  days 

of  the  completion  of  the  live  fire  test.  The  report 
must  include  the  firing  results,  test  conditions, 
limitations,  and  conclusions  and  be  submitted  in  both 

classified  and  unclassified  form. 

o  A  separate  test  report  will  be  produced  by  OSD.  The 

conclusions  of  the  OSD  report  will  be  independent 
of,  and  uncoordinated  with,  the  conclusions  of  the 

Service  report. 

o  Congress  shall  have  access  to  all  live  fire  test  data 
and  a T T  live  fire  test  reports  held  by  or  produced 
by  the  Secretary  of  the  Service  concerned  or  by  OSD. 

o  The  costs  of  all  live  fire  tests  shall  be  paid  from 

funding  for  the  system  being  tested.  In  some  instances, 
the  DDDR&E  (LFT)  may  elect  to  supplement  such  funds 
for  the  acquisition  of  targets  or  target  simulators, 
although  the  ultimate  responsibility  rests  on  the 

Service  concerned. 

19.3  TESTING  FOR  NUCLEAR  HARDNESS  AND  SURVIVABILITY 

19.3.1  Background 

Nuclear  survivability  must  be  incorporated  into  the  design, 
acquisition,  and  operation  of  all  systems  that  must  perform  critical 
missions  in  a  nuclear  environment.  Nuclear  survivability  is  achieved 
through  a  combination  of  four  methods:  hardness,  avoidance,  proliferation, 
and  reconstitution.  Hardness  allows  a  system  to  physically  withstand 
a  nuclear  attack.  Avoidance  encompasses  the  measures  taken  to  not 
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encounter  a  nuclear  environment.  Proliferation  Involves  having  sufficient 
numbers  of  a  system  to  compensate  for  probable  losses.  Reconstitution 
includes  the  actions  taken  to  repair  or  resupply  damaged  units  in  time 
to  complete  a  mission  satisfactorily. 

There  is  a  wide  variety  of  possible  effects  from  a  nuclear 
detonation.  They  include:  electromagnetic  pulse  (EMP),  ionizing 
radiation,  thermal  radiation,  blast,  shock,  dust,  debris,  blackout,  and 
scintillation.  Each  weapon  system  is  susceptible  to  some,  but  not  all, 
of  these  effects,  The  Program  Manager  and  his  staff  must  identify  the 
effects  that  may  have  an  impact  on  the  system  under  development  and  manage 
the  design,  development,  and  testing  of  the  system  in  a  manner  that 
minimizes  degradation.  The  variety  of  possible  nuclear  effects  is 
described  more  fully  in  the  "Nuclear  Survivability  Handbook  for  Air  Force 
OT&E"  (Referenced). 

19.3.2  Assessing  Nuclear  Survivability  Throughout  The  System 

Acquisition  Cycle 

The  Program  Manager  must  ensure  that  nuclear  survivability 
issues  are  addressed  throughout  the  system  acquisition  cycler-  During 
the  Concept  Exploration/Definition  Phase,  the  survivability  requirements 
stated  in  the  various  Service  requirements  documents  (e.g.,  Required 
Operational  Capability  (ROC),  Statement  of  Operational  Need  (SON),  or 
Operational  Requirement  (OR))  should  be  verified,  refined,  or  further 
defined.  During  the  Concept  Demonstration  and  Validation  Phase,  trade-offs 
between  hardness  levels  and  other  system  characteristics  (such  as;  Weight, 
speed,  range,  cost  and  operational  effectiveness)  should  be  described 
quantitatively.  Trade-offs  between  hardness,  avoidance,  proliferation, 
and  reconstitution  as  methods  for  achieving  survivability  should  also 
be  considered  at  this  time.  During  the  Full-Scale  Development  ^Phase, 
the  system  must  be  adequately  tested  to  confirm  that  hardness  objectives, 
criteria,  requirements ,  and  specifications  are  met.  Plans  for  nuclear 
hardness  and  survivability  testing  must  be  outlined  in  the  Test  and 
Evaluation  Master  Plan.  The  appropriate  commands  must  make  provision 
for  test  and  hardness  surveillance  equipment  and  procedures  so  that 
required  hardness  levels  can  be  maintained  once  the  system  is  operational. 

During  the  Production  Phase,  system  hardness  is  maintained 
through  an  active  hardness  assurance  program.  Such  a  program  ensures 
that  the  end  product  conforms  to  hardness  design  specifications  and  that 
hardness  aspects  are  re-evaluated  before  any  retrofit  changes  are  made 
to  existing  systems. 

Once  a  system  is  operational,  a  hardness  surveillance  program 
may  be  implemented  to  maintain  system  hardness  and  to  identify  any  further 
evaluation,  testing,  or  retrofit  changes  required  to  ensure  survivability. 
A  hardness  surveillance  program  consists  of  a  set  of  scheduled  tests 


and  inspections  to  ensure  that  a  system's  designed  hardness  is  not  degraded 
through  operational  use,  logistic  support,  maintenance  actions,  or  natural 
causes. 

19.3.3  Test  Planning 

The  "Nuclear  Survivability  Handbook  for  Air  Force  OT&E"  describes 
the  following  challenges  associated  with  nuclear  hardness  and  survivability 
testing: 

(1)  The  magnitude  and  range  of  effects  from  a  nuclear 
burst  are  much  greater  than  those  from  conventional 
explosions  that  may  be  used  to  simulate  nuclear  bursts. 

Nuclear  detonations  have  effects  not  found  in 
conventional  explosions.  The  intense  nuclear 
radiation,  blast,  shock,  thermal,  and  EMP  fields 
are  difficult  to  simulate.  In  addition,  systems 
are  often  tested  at  stress  levels  that  are  either 
lower  than  those  established  by  the  criteria  or  lower 
than  the  level  needed  to  cause  damage  to  the  system. 

(2)  The  yields  and  configurations  for  underground 
testing  are  limited.  It  is  generally  not  possible 
to  test  all  relevant  effects  simultaneously  or  to 
observe  possibly  important  synergisms  between  effects. 

(3)  System-level  testing  for  nuclear  effects  is 
normally  expensive,  takes  years  to  plan  and  conduct, 
and  requires  specialized  expertise.  Often,  classes 
of  tests  conducted  early  in  the  program  are  not 
repeated  later.  Therefore,  operational  requirements 
should  be  folded  into  these  tests  from  the  start, 
often  early  in  the  acquisition  process.  This  mandates 
a  more  extensive  combined  DT&E/OT&E  test  program 
than  normally  found  in  other  types  of  testing. 

Program  Managers  and  test  managers  must  remain  sensitive  to 
the  ambiguities  involved  in  testing  for  nuclear  survivability.  For 
example,  there  is  no  universal  quantitative  measure  of  survivability, 
and  statements  of  survivabili ty  may  lend  themselves  to  a  variety  of 
interpretations.  Moreover,  it  can  be  difficult  to  combine  system 
vulnerability  estimates  for  various  nuclear  effects  into  an  assessment 
of  overall  survivability.  As  a  result,  program/test  managers  must  exercise 
caution  when  developing  test  objectives  and  specifying  measures  of  merit 
related  to  nuclear  survivability. 
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19.3.4  T3St  Execution 


For  nuclear  hardness  and  survivability  testing,  DT  and  OT  efforts 
are  often  combined  because  it  is  not  possible  to  test  in  an  actual 
operational  nuclear  environment.  The  use  of  an  integrated  DT/OT  program 
requires  early  and  continuous  dialogue  between  the  two  test  communities 
so  that  each  understands  the  needs  of  the  other  and  to  obtain  the  maximum 
cooperation  in  meeting  objectives. 

Test  and  evaluation  techniques  available  to  validate  the  nuclear 
survivability  aspects  of  systems  and  subsystems  include  underground  nuclear 
testing,  environmental  simulation  (system-level,  subsystem  level,  and 
component  level),  and  analytical  simulation.  Table  19-4  outlines  the 
major  activities  relevant  to  the  assessment  of  nuclear  hardness  and 
survivability  and  the  phases  of  the  acquisition  cycle  in  which  they  occur. 

19.4  SUMMARY 

The  vulnerability  and  lethality  aspects  of  a  system  can  be 
evaluated  through  live  fire  tests.  These  tests  are  used  to  provide  an 
insight  into  the  system's  ability  to  protect  its  crew  and  to  continue 
to  operate/fight  after  being  hit  by  enemy  weapons.  It  provides  a  means 
for  examining  the  damages  inflicted,  not  only  on  material,  but  also  on 
personnel.  Live  fire  testing  also  provides  an  opportunity  to  assess 
the  effects  of  complex  environments  that  crews  are  likely  to  encounter 
in  combat. 


Nuclear  survivability  must  be  carefully  evaluated  during  the 
system  acquisition  cycle.  Trade-offs  between  hardness  levels  and  other 
system  characteristics,  such  as  weight,  speed,  range,  cost,  etc.,  must 
be  evaluated.  Nuclear  survivability  testing  is  difficult,  and  the 
evaluation  of  test  results  may  lend  themselves  to  a  variety  of 
interpretations.  Therefore,  Program  Managers  must  exercise  caution  when 
developing  test  objectives  related  to  nuclear  survivability. 
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Table  19-4.  Nuclear  Hardness  and  Survivability  Assessment  Activities 


•  Preparation  of  Test  and  Evaluation  Master  Plan  (TEMP)  that 
includes  initial  plans  for  nuclear  hardness  and  survivability 

(NH&S)  tests 

-  Identification  of  NH&S  requirements  in  verifiable  terms 

-  Identification  of  special  NH&S  test  facility  requirements 
with  emphasis  on  long-lead-time  items 

•  Development  of  nuclear  criteria 


•  Increased  test  planning 

•  TEMP  update 

•  Conduct  of  NH&S  trade  s  udies 

-  NH&S  requirements  versus  other  system  requirements 

-  Alternate  methods  for  achieving  NH&S 

•  Conduct  of  limited  testing 

-  Piece-part  hardness  testing 

-  Design  concept  trade-off  testing 

-  Technology  demonstration  testing 

•  Development  of  system  specifications  that  include  quantitative 
hardness  levels 
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Table  19-4.  Nuclear  Hardness  and  Survivability 
Assessment  Activities  (Concluded) 


•  First  opportunity  to  test  prototype  hardware 

•  TEMP  update 

•  Development  of  Nuclear  Hardness  Design  Handbook 

-  Prior  to  Preliminary  Design  Review 

-  Usually  prepared  by  nuclear  effects  specialty  contractor 

•  Conduct  of  testing 

-  Pre-Critical  Design  Review  (CDR)  development  and 
qualification  tests 

-  Development  testing  on  nuclear-hardened  piece  parts, 
materials,  cabling,  and  circuits 

-  NH&S  box  and  subsystem  qualification  tests  (post-CDR) 

-  Acceptance  tests  to  verify  that  hardware  meets 
specifications  (post-CDR,  prior  to  first  delivery) 

-  System-level  hardness  analysis  (using  box  and  subsystem 
test  results) 

-  System-level  NH&S  test 


•  Implementation  of  program  to  ensure  system  retains  its  NH&S 
properties  throughout  production  and  deployment 

•  Screening  of  production  hardware  for  hardness 


•  Implementation  by  user  of  procedures  to  ensure  that  system's 
operation,  logistic  support  and  maintenance  do  not  degrade 
hardness  features 

•  Reassessment  of  survivability  throughtout  system  life  cycle 
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CHAPTER  20 
MULTISERVICE  TESTS 


20.1  INTRODUCTION 

This  chapter  discusses  the  planning  and  management  of  a 
multi  service  test  program.  A  multi  service  test  program  is  conducted 

when  a  system  is  to  be  acquired  for  use  by  more  than  one  Service,  or 

when  a  system  must  interface  with  equipment  of  another  Service.  A 

multiservice  test  program  should  not  be  confused  with  the  OSD-sponsored, 
non-acquisition-oriented  Joint  Test  and  Evaluation  (JT&E)  program.  A 
brief  description  of  the  JT&E  program  is  provided  in  Chapter  3. 

20.2  BACKGROUND 

DOD  Directive  5000.3  contains  a  definition  of  multiservice 
test  and  evaluation,  designates  the  participants  in  the  program,  and 

gives  a  Lead  Service  responsibility  for  preparing  a  single  report 
concerning  a  system's  operational  effectiveness  and  suitability.  (The 
Lead  Service  is  the  Service  responsible  for  the  overall  management  of 
a  multi  service  program.  A  "Supporting  Service"  is  a  Service  designated 
to  assist  the  Lead  Service.) 

A  multi  service  test  and  evaluation  program  may  include  either 
DT&E  or  OT&E  or  both.  The  Services  Operational  Test  Agencies  have  executed 
a  formal  Memorandum  of  Agreement  on  multi  service  OT&E  (Reference  35) 
that  provides  a  framework  for  the  conduct  of  a  multiservice  operational 
test  program. 

Air  Force  Regulation  80-14  describes  the  procedures  followed 
in  a  multi  service  T&E  program  as  follows: 

(1)  In  a  multiservice  acquisition  program,  T&E  is 
planned  and  conducted  according  to  Lead  Service 
regulations.  The  designated  Lead  Service  will  have 
the  overall  responsibility  for  management  of  the 
multi  service  program  and  will  ensure  that  supporting 
service  requirements  are  included.  If  another  Service 
has  certain  unique  T&E  requirements,  testing  for 
these  unique  requirements  may  be  planned,  funded, 
and  conducted  according  to  that  Service's  regulations. 

(2)  Participating  Services  will  prepare  reports 
in  accordance  with  their  respective  regulations. 

The  Lead  Service  will  prepare  and  coordinate  a  single 
DT&E  report  and  a  single  OT&E  report,  which  will 
summarize  the  conclusions  and  recommendations  of 
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each  Service's  reports.  Rationale  will  be  provided 
to  explain  any  significant  differences.  The  individual 
Service  reports  will  be  attached  to  this  single  report. 

(3)  Deviations  from  the  Lead  Service  T&E  regulations 
may  be  accommodated  by  mutual  agreement  among  the 
Services  involved. 

20.3  TEST  PROGRAM  RESPONSIBILITIES 

The  Lead  Service  has  overall  management  responsibility  for 
the  program.  It  must  ensure  that  supporting  Service  requirements  are 
included  in  the  formulation  of  the  basic  resource  and  planning  documents. 

A  Test  Management  Council  (TMC)  is  established  for  each 
multiservice  test  program.  Its  membership  consists  of  one  senior 
representative  from  each  participating  Service  or  agency  headquarters. 
The  TMC  works  very  closely  with  the  PMO  and  is  responsible  for  arbitrating 
all  disagreements  among  Services  that  cannot  be  resolved  at  the  working 
level . 


Resource  requirements  are  documented  in  the  Test  and  Evaluation 
Master  Plan.  Each  participating  Service  is  directed  to  budget  for  the 
testing  necessary  to  accomplish  its  assigned  test  objectives  and  for 
the  participation  of  its  personnel  and  equipment  in  the  entire  test 
program. 


20.4  TEST  TEAM  STRUCTURE 

A  sample  test  team  structure  is  shown  in  Figure  20-1.  As  shown 
in  the  figure.  Service  test  teams  work  through  a  Service  Deputy  Test 
Director,  or  senior  representati ve.  The  Test  Director  exercises  test 
management  authority  over  the  test  teams,  but  not  operational  control. 
His  responsibilities  include  integration  of  test  requirements  and  efficient 
scheduling  of  test  events.  The  Deputy  Test  Directors  exercise  operational 
control  or  test  management  authority  over  their  Service  test  teams  in 
accordance  with  their  Service  directives.  Additionally,  they  act  as 
advisors  to  the  Test  Director;  represent  their  Service's  interests;  and 
are  responsible,  at  least  administratively,  for  resources  and  personnel 
provided  by  their  Services. 

20.5  TEST  PLANNING 

Test  planning  for  multi  service  T&E  is  accomplished  in  the  manner 
prescribed  by  Lead  Service  directions  and  in  accordance  with  the  following 
general  procedures  extracted  from  the  "Memorandum  of  Agreement  on 
Multi-Service  0 T&E  and  Joint  T&E": 
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*7-2-MCL2 -0001 85-05  i.  used  for  complex  programs  with  many  participants 


SOURCE:  "MEMORANDUM  OF  AGREEMENT  ON  MULTI-SERVICE  OT&E  AND  JOINT  T&E~ 


Figure  20-1.  Simple  Multiservice  0T&E  Test  Team  Composition 
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(1)  The  Lead  Service  T&E  agencies  begin  the  planning  process 
by  issuing  a  call  to  the  supporting  Service  T&E  agencies  for  critical 
issues  and  test  objectives. 

(2)  The  Lead  Service  T&E  agencies  consolidates  the  objectives 
into  a  list  and  coordinates  the  list  with  the  supporting  Service  T&E 
agencies. 


(3)  The  Lead  Service  T&E  agency  accommodates  supporting  Service 
T&E  requirements  and  inputs  in  the  formal  coordination  action  of  the 
TEMP. 

(4)  Participating  T&E  agency  project  officers  assign 
responsibility  for  the  accomplishment  of  test  objectives  (from  the 
consolidated  list)  to  each  T&E  agency.  These  assignments  are  made  in 
a  mutually  agreeable  manner.  Each  agency  is  then  responsible  for  resource 
identification  and  accomplishment  of  its  assigned  test  objectives  under 
the  direction  of  the  Lead  Service  T&E  agency. 

(5)  Each  participating  agency  prepares  the  portion  of  the 
overall  test  plan(s)  for  its  assigned  objectives,  in  the  Lead  Service's 
test  plan(s)  format,  and  identifies  its  data  needs. 

(6)  The  Lead  Service's  T&E  agency  prepares  the  multi  service 
T&E  test  plan(s),  consolidating  the  inputs  from  all  participating  agencies. 

20.6  DISCREPANCY  REPORTING 

In  a  multi  service  T&E  program,  a  discrepancy  report  is  a  report 
of  any  condition  which  reflects  adversely  on  the  item  being  tested  and 
which  must  be  reported  outside  the  test  team  for  corrective  action. 
The  discrepancy  reporting  system  of  the  Lead  Service  is  normally  used. 
All  members  of  the  multiservice  test  team  will  report  discrepancies  through 
their  Service's  system. 

Items  undergoing  test  will  not  necessarily  be  used  by  each 
of  the  Services  for  identical  purposes.  As  a  result,  a  discrepancy 
considered  disqualifying  by  one  Service  is  not  necessari ly  disqualifying 
for  all  of  the  Services.  Discrepancy  reports  of  a  disqualifying  nature 
must  include  a  statement  by  the  concerned  Service  of  why  the  discrepancy 
has  been  so  classified.  It  also  includes  statements  by  the  other  Services 
as  to  whether  or  not  the  discrepancy  significantly  affects  them. 

In  the  event  that  one  of  the  participating  Services  identifies 
a  discrepancy  that  it  considers  warrants  termination  of  the  test,  the 
circumstances  are  reported  immediately  to  the  Test  Director. 
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20.7  TEST  REPORTING 

The  following  test  reporting  policy  applies  to  multi  service 
OT&E  programs: 

(1)  Interim  test  reports  are  not  normally  prepared.  If  they 
are  required  on  a  particular  program,  they  are  prepared  in  accordance 
with  Lead  Service's  directives  and  coordinated  with  all  participating 
OT&E  agencies  prior  to  release. 

(2)  Within  60  days  of  the  end  of  testing,  the  multi  service 
OT&E  team  must  present  a  factual  report  of  the  test  to  all  participating 
OT&E  agencies.  (This  factual  report  presents  the  data  collected,  but 
no  evaluation,  conclusions,  or  recommendations  concerning  the  data.) 

(3)  Each  participating  OT&E  agency  prepares  an  independent 
evaluation  report  in  its  own  format  and  forwards  that  report  through 
its  normal  Service  channels. 

(4)  Approved  independent  evaluation  reports  are  distributed 
to  all  participating  OT&E  agencies. 

(5)  The  Lead  Service  OT&E  agency  is  responsible  for  preparing 
the  Defense  Acquisition  Board  (DAB)  briefing(s)  which  is  (are)  coordinated 
with  all  participating  OT&E  agencies. 

20.8  SUMMARY 

Multi  service  test  programs  are  conducted  by  two  or  more  Services 
when  a  system  is  to  be  acquired  by  more  than  one  Service  or  when  a  system 
must  interface  with  equipment  of  another  Service.  Test  procedures  for 
multiservice  test  and  evaluation  follow  those  of  the  designated  Lead 
Service  with  mutual  agreements  resolving  areas  where  deviations  are 
necessary.  Care  must  be  exercised  when  integrating  test  results  and 
reporting  discrepancies  since  items  undergoing  test  may  be  used  for 
different  purposes  in  different  Services.  Close  coordination  is  required 
to  ensure  that  an  accurate  summary  of  the  developing  system's  capabilities 
is  provided  to  Service  and  DoD  decision  authorities. 
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CHAPTER  21 

INTERNATIONAL  TEST  AND  EVALUATION  PROGRAMS 


21.1  INTRODUCTION 

This  chapter  discusses  test  and  evaluation  from  an  international 
perspective.  It  describes  the  OSD-sponsored  Foreign  Weapons  Evaluation 
(FWE)  and  NATO  Comparative  Test  Programs  and  discusses  factors  that  bear 
on  the  test  and  evaluation  of  multinational  acquisition  programs. 

21.2  FOREIGN  WEAPONS  EVALUATION  PROGRAM 

21.2.1  Program  Objective 

The  Foreign  Weapons  Evaluation  Program  is  designed  to  support 
the  evaluation  of  a  foreign  nation's  weapons  system,  equipment,  or 
technology  in  terms  of  its  potential  to  meet  a  valid  requirement  of  one 
or  more  of  the  U.S.  Armed  Services.  Additional  goals  of  the  FWE  program 
include  avoiding  unnecessary  duplication  in  development,  enhancing 
standardization  and  interoperability,  and  promoting  international  technology 
exchanges.  The  FWE  program  is  not  intended  for  use  in  exploiting  threat 
systems  or  for  intelligence  gathering  purposes.  The  primary  objective 
of  the  program  is  to  reduce  the  costs  of  research  and  development,  while 
leading  to  the  acquisition  of  foreign  equipment  for  U.S.  use.  Policy 
and  procedures  for  the  execution  of  the  FWE  program  are  documented  in 
DoD  5000. 3-M- 2. 

21.2.2  Program  Administration 

Foreign  weapons  evaluation  activities  and  responsibilities  are 
assigned  to  the  Director  Defense  Test  and  Evaluation  (now  Deputy  Director 
Defense  Research  and  Engineering  (Test  and  Evaluation)  (DDDR&E(T&E) )  by 
direction  of  the  Congress  in  1980.  Each  year,  sponsoring  military  services 
forward  to  the  DDDR&E(T&E)  Candidate  Nomination  Proposals  (CNPs)  for  systems 
to  be  evaluated  under  the  FWE  program.  The  services  are  encouraged  to 
prepare  and  submit  a  CNP  whenever  a  promising  candidate  is  found  that 
appears  to  satisfy  a  current  or  potential  service  requirement.  A  CNP 
must  contain  the  information  as  required  by  DoD  5000.3-M-2. 

The  fundamental  criterion  for  FWE  program  selection  is  the 
candidate  system's  potential  to  satisfy  an  existing  or  projected  operational 
or  training  requirement  or  its  possible  contribution  to  the  U.S.  technology 
base.  Additional  factors  influencing  candidate  selection  include  the 
following:  candidate  maturity,  available  test  data,  multiservice  interest, 
existence  of  a  statement  of  operational  requirement  need,  potential  for 
subsequent  procurement,  sponsorship  by  U.S.  based  licensee,  realistic 
evaluation  schedule  cost,  DoD  Component  OSD  evaluation  cost  sharing  proposal 
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and  preprogrammed  procurement  funds.  For  technology  evaluation  programs, 
within  the  FWE  program,  the  candidate  nomination  proposal  must  address 
the  specific  arrangements  under  which  the  U.S.  and  the  foreign  participants 
(governments,  armed  forces,  corporations)  will  operate.  These  may  include 
government-to-government  Memoranda  of  Agreement,  private  industry  licensing 
agreements,  data  exchange  agreements  and/or  cooperative  technology  exchange 
programs. 


Foreign  weapons  evaluation  activities  are  funded  by  OSD  and 
executed  by  the  Service  with  the  potential  need  for  the  system.  Points 
of  contact  at  the  headquarters  level  in  each  of  the  Services  monitor  the 
conduct  of  the  programs.  Work  is  performed  in  laboratories  and  test  centers 
throughout  the  country.  Systems  evaluated  recently  under  the  FWE  program 
include  millimeter  wave  communications  equipment,  chemical  defense 
equipment,  gunnery  devices,  maritime  decoys,  and  navigational  systems. 

21.3  NATO  COMPARATIVE  TEST  PROGRAM 

The  NATO  Comparative  Test  Program  is  an  expanded  version  of 
the  FWE  program.  It  was  created  by  an  act  of  Congress  in  the  FY86  Defense 
Authorization  Bill.  The  program  supports  the  evaluation  of  NATO  nations' 
weapons  systems,  equipment,  and  technology  and  assesses  their  suitability 
for  use  by  U.S.  forces.  The  selection  criteria  for  the  NATO  Comparative 
Test  Program  are  essentially  the  same  as  for  the  FWE  program  with  the 
exception  that  the  equipment  must  be  produced  by  a  NATO  member  nation 
and  be  considered  as  an  alternative  to  a  system  either  in  the  late  stage 
of  development  in  the  US  or  offer  a  cost,  schedule,  or  performance  advantage 
over  US  equipment.  In  addition,  the  NATO  Comparative  Test  Program  requires 
that  notification  be  sent  to  the  Armed  Services  and  Appropriations 
Committees  of  the  House  of  Representatives  and  Senate  before  funds  are 
obligated.  With  this  exception,  the  NATO  Comparative  Test  Program  follows 
the  same  nomination  process  and  administrative  procedures  as  the  Foreign 
Weapons  Evaluation  Program.  Guidelines  for  the  program  will  also  be 
contained  in  DoD  5000.3-M-2. 

Proposals  recently  funded  under  the  NATO  Comparative  Test  Program 
include  test  and  evaluation  of  a  German  mine  reconnaissance  and  detection 
system  for  the  Army,  a  United  Kingdom-designed  minehunter  for  the  Navy, 
and  the  Norwegian  Penguin  missile  system  for  the  Air  Force.  According 
to  the  FY88  Report  of  the  Secretary  of  Defense  to  the  Congress,  the  program 
has  generated  considerable  interest  among  the  NATO  allied  nations  and 
has  become  a  primary  means  of  promoting  armaments  cooperation  within  NATO. 

Problems  associated  with  testing  foreign  weapons  normally  stem 
from  politics,  national  pride,  and  a  lack  of  previous  test  data.  When 
foreign  companies  introduce  weapon  systems  for  test,  they  will  often  attempt 
to  align  the  U.S.  military/Congressional  organizations  with  their  systems. 
For  example,  when  a  foreign  nation  introduced  an  anti-tank  weapon  to  the 
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Army,  they  did  so  by  having  a  U.S.  Senator  write  the  Army  stating  a  need 
for  the  system.  The  letter  had  attached  a  document  containing  doctrine 
to  employ  the  system  and  a  test  concept  for  use  when  evaluating  the  system. 
Systems  that  are  tested  in  the  NATO  Comparative  Test  Program  often  become 
involved  in  national  pride.  The  test  community  must  be  careful  not  to 
allow  national  pride  to  be  a  driving  force  in  the  evaluation.  The  9mm 
pistol  competition  in  NATO,  at  times,  took  on  the  form  of  an  international 
soccer  match,  with  each  competing  nation  cheering  for  their  pistol  and 
many  other  nations  selecting  sides.  The  evaluation  of  the  9mm  pistol 
was  difficult  because  of  these  forces.  United  States  testers  must  make 
every  effort  to  obtain  all  available  test  data  on  foreign  systems.  These 
data  can  be  used  to  help  validate  the  evolving  test  data  and  additional 
test  data  during  the  evaluation. 

21.4  TEST  AND  EVALUATION  MANAGEMENT  IN  MULTINATIONAL  PROGRAMS 

Rationalization,  standardization,  and  interoperability  have 
become  increasingly  important  elements  in  the  materiel  acquisition  process. 
Public  Law  94-361,  passed  on  July  14,  1976,  requires  that  "equipment  for 
use  of  personnel  of  the  Armed  Forces  of  the  United  States  stationed  in 
Europe  under  the  terms  of  the  North  Atlantic  Treaty  should  be  standardized 
or  at  least  interoperable  with  equipment  of  other  members  of  the  North 
Atlantic  Treaty  Organization"  (Reference  4,  pages  1-2).  Program  Managers 
and  test  managers  must,  therefore,  be  fully  aware  of  any  potential 
international  applications  of  the  systems  for  which  they  are  responsible. 
The  Joint  Logistics  Commanders  Guide  for  the  Management  of  Multinational 
Programs  published  by  the  Defense  Systems  Management  College  ( Reference 
47)  is  a  valuable  compendium  of  information  for  the  Program  Manager  of 
a  developing  system  with  potential  multinational  applications. 

Representatives  of  the  United  States,  United  Kingdom,  France, 
and  Germany  have  signed  a  Memorandum  of  Agreement  concerning  the  mutual 
acceptability  of  each  country's  test  and  evaluation  data.  This  agreement 
seeks  to  avoid  redundant  testing  by  documenting  the  extent  of  understanding 
between  the  governments  involved  concerning  the  mutual  acceptability  of 
their  respective  T&E  procedures  for  those  systems  that  are  developed  in 
one  country  and  are  candidates  for  procurement  by  one  or  more  of  the  other 
countries.  Focal  points  for  both  development  and  operational  testing 
in  each  of  the  countries  are  identified,  and  procedures  governing  the 
generation  and  release  of  T&E  data  are  described  in  the  MOU. 

Early  and  thorough  planning  is  an  important  element  of  any 
successful  test  and  evaluation  program  but  is  even  more  critical  in  a 
multinational  program.  Agreement  must  be  reached  concerning  test  and 
evaluation  procedures,  data  requirements  and  methodology.  Differences 
in  tactics,  battlefield  representations,  and  military  organizations  may 
also  make  it  difficult  for  one  nation  to  accept  another's  test  data. 
Therefore,  agreement  must  be  reached  in  advance  concerning  the  operational 
test  scenario  and  battlefield  representation  that  will  be  used. 
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U.S.  AND  NATO  ACQUISITION  PROGRAMS 


Some  test  programs  involve  combined  development  and  test  of 
new  weapon  systems  for  U.S.  and  other  NATO  countries.  In  these  programs, 
some  differences  from  the  regular  "way  of  doing  things"  occurs.  For 
example,  the  formulation  of  the  Request  for  Proposal  (RFP)  must  be 
coordinated  with  the  North  Atlantic  Program  Management  Agency  (NAPMA) 
and  their  Inputs  to  the  statement  of  work,  data  requirements,  operational 
test  planning,  and  test  schedule  formulation  must  be  included.  Also, 
their  operational  user.  Force  Command  must  be  involved  in  the  operational 
test  program.  Usually,  a  Multinational  Memorandum  of  Understanding  (MMOU) 
is  created  concerning  test  program  and  production  funding,  test  resources, 
test  team  composition,  use  of  national  assets  for  testing,  etc.  Nations 
are  encouraged  to  use  the  data  that  another  nation  has  gathered  on  similar 
test  programs  to  avoid  duplication  of  effort. 

For  example,  during  the  U.S.  and  NATO  AWACS  Electronic 
Support  Measures  (ESM)  program ,  both  U.S.  and  NATO 
E-3A's  will  be  used  for  test  aircraft  in  combined 
DT&E  testing  and  the  subsequent  OT&E  testing.  Testing 
will  be  conducted  in  the  U.S.  and  in  the  European 
threaters.  The  Joint  Test  Force  will  be  composed 
of  Program  Management  Office,  contractor,  U.S. 
operational  users,  AFOTEC,  Force  Command  (NATO  users) 
and  logistics  personnel  for  this  program.  A 
Multinational  Memorandum  of  Agreement  for  this  program 
was  created.  The  U.S.  program  is  managed  by  the 
AWACS  System  Program  Office  and  the  NATO  program  is 
managed  by  the  North  Atlantic  Program  Management  Agency 
(NAPMA). 

21.6  SUMMARY 

The  procurement  of  weapon  systems  for  use  by  the  US  Armed  Forces 
from  foreign  nations  can  provide  the  following  advantages:  reduced  research 
and  development  costs;  a  faster  initial  operational  capability;  improved 
interoperability  with  friendly  nations;  and  lower  procurement  costs  because 
of  economies  of  scale.  The  testing  of  such  systems  presents  specific 
challenges  in  order  to  accommodate  the  needs  of  all  users.  Such  testing 
requires  careful  advance  planning  and  systematic  execution.  Expectations 
and  understandings  must  be  well  documented  at  an  early  stage  to  ensure 
that  the  test  results  have  utility  for  all  concerned. 
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CHAPTER  22 

T8£  POLICY  STRUCTURE  AND  OVERSIGHT  MECHANISM 


22.1  INTRODUCTION 

This  chapter  provides  an  overview  of  the  policy  and  organizations 
which  govern  the  conduct  of  test  and  evaluation  activities  within  the 
Department  of  Defense  (DoD).  It  discusses  Congressional  legislation  and 
activities  for  compliance  by  the  DoD.  It  outlines  the  responsibilities 
of  the  DoD  test  organizations,  at  both  the  OSD  and  Service  level,  and 
describes  related  T&E  policy. 

22.2  THE  CONGRESS 

The  Congress  has  shown  a  long-standing  interest  in  influencing 
the  DoD  acquisition  process.  In  the  early  seventies,  in  response  to  urging 
by  Congress  and  recommendations  by  a  Presidential  Blue  Ribbon  Panel  on 
Defense  Management,  the  Deputy  Secretary  of  Defense,  David  Packard, 
promulgated  a  package  of  policy  initiatives  which  established  the  Defense 
Systems  Acquisition  Review  Council  (DSARC).  The  DSARC  was  organized  to 
resolve  acquisition  issues,  whenever  possible,  and  to  provide 
recommendations  to  the  Secretary  of  Defense  (SECDEF)  on  the  acquisition 
of  major  weapon  systems.  Also  as  a  result  of  the  Congressional  Directives, 
the  Army  and  Air  Force  established  independent  operational  test  agencies. 
The  Navy  Operational  Test  and  Evaluation  Force  came  into  being  in  the 
late  sixties.  In  1983,  similar  concerns  led  Congress  to  direct  the 
establishment  of  the  independent  office  of  Director  Operational  Test  and 
Evaluation  (DOT&E)  within  the  OSD.  In  1985  a  report  released  by  the 
President's  Blue  Ribbon  Commission  on  Defense  Management,  chaired  by  David 
Packard,  made  significant  recommendations  on  the  management  and  oversight 
of  DoD's  acquisition  process  and  specifically  test  and  evaluation.  All 
of  the  Commission's  recommendation  have  not  been  implemented  and  the  full 
impact  of  these  recommendations  is  not  yet  realized.  In  FY87  the  Defense 
Authorization  Act  required  live  firing  testing  of  weapon  systems  before 
the  production  phase  begins. 

The  Department  of  Defense  is  required  to  provide  to  the  Congress 
the  following  reports  on  test  and  evaluation: 

o  Congressional  Data  Sheets  (CDS).  The  CDS  is  an  annual 
report  on  each  major  system  acquisition.  It  must  be  updated 
before  the  contract  is  awarded  and  when  procurement  of  the  system 
is  requested  in  the  fiscal  year.  The  CDS  describes  the  DT&E 
and  OT&E  to  be  performed  and  the  systems  characteristics. 


22-1 


o  Selected  Acquisition  Report  (SAR).  The  SAR  describes  the 
system  characteristics  required  and  outlines  significant  progress 
and  problems  encountered.  It  lists  tests  completed  and  issues 
identified  during  testing. 

o  Annual  System  Operational  Test  Report.  The  Annual  Systems 
Operational  Test  Report  is  provided  by  the  DOT&E  to  the  SECDEF 
and  the  Committees  on  Armed  Services  and  Appropriations.  The 
report  provides  a  narrative  and  resource  summary  of  OT&E  and 
OT&E-related  issues,  activities,  and  assessments. 

o  Low  Rate  Initial  Production  Report  (LRIP).  Before  proceeding 
beyond  LRIP  for  each  major  system  acquisition  program,  the 
Director,  Operational  Test  &  Evaluation  must  report  to  the  SECDEF 
and  Congress  on  the  adequacy  of  OT&E  and  whether  the  T&E  results 
confirm  that  the  item  or  component  actually  tested  are  effective 
and  suitable  for  combat. 

22.3  OSD  OVERSIGHT  STRUCTURE 

The  organization  of  the  Department  of  Defense  for  the  oversight 
of  test  and  evaluation  is  illustrated  in  Figure  22-1.  The  oversight  of 
test  and  evaluation,  in  the  Office  of  the  Secretary  of  Defense  (OSD), 
is  performed  by  two  primary  offices:  the  Deputy  Director,  Defense  Research 
and  Engineering  (Test  and  Evaluation)  (DDDRE(T&E) ) ,  and  the  Director 
Operational  Test  and  Evaluation  (DOT&E).  The  management  of  acquisition 
programs  in  OSD  is  performed  by  the  Defense  Acquisition  Executive  (DAE) 
who  uses  the  Defense  Acquisition  Board  and  ten  committees  to  process 
information  for  decisions.  The  Under  Secretary  of  Defense  (Acquisition) 
(USD(A))  is  the  DAE  and  he  uses  the  DAB  and  its  committees  to  provide 
the  senior  level  decision  process  for  the  acquisition  of  weapon  systems. 

22.3.1  Defense  Acquisition  Executive  (DAE) 

The  Defense  Acquisition  Executive  position  was  established  in 
September  1986.  He  is  the  USD(A)  and  his  responsibilities  include 
“establishing  policies  for  acquisition  (including  procurement,  research 
and  development,  logisitics,  development  testing,  and  contracts 
administration)  for  all  elements  of  the  Department  of  Defense.  His  charter 
includes  the  authority  over  the  Service  and  Defense  Agencies  on  policy, 
procedure  and  execution  of  the  acquisition  process. 

22.3.2  Defense  Acquisition  Board  (DAB) 

The  DAB  is  the  primary  forum  used  by  OSD  to  provide  advice, 
assistance  and  recommendations,  and  to  resolve  issues  regarding  all 
operating  and  policy  aspects  of  the  DoD  Acquisition  System.  The  DAB  is 
the  senior  management  acquisition  board  chaired  by  the  DAE  and  attended 
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Figure  22-1.  OSD  Oversight  of  T&E 


by  the  Vice  Chairman  of  Joint  Chiefs  of  Staff,  Principal  Under  Secretary 
of  Defense  for  Acquisition  and  the  Services  Acquisition  Executives.  As 
illustrated  in  Figure  22-2  the  DAB  conducts  business  through  ten  working 
committees.  (DoDD  5000.49) 

22.3.3  Test  and  Evaluation  Committee  (TEC) 

The  DAB  committee  that  is  responsible  for  T&E  is  the  Test  and 
Evaluation  Committee.  As  illustrated  in  Figure  22-2,  the  TEC  has  the 
responsibility  of  DT&E,  OT&E  and  test  facilities.  The  TEC  holds  pre-DAB 
meetings  for  the  purpose  of  resolving  resourcing  issues,  developing 
recommendations  and  highlighting  significant  issues  to  be  addressed  by 
the  DAB.  The  TEC  is  chaired  by  the  DOT&E  and  has  representation  from 
DDDRE (T&E )  and  the  Service  Acquisition  Executives. 

22.3.4  Defense  Resources  Board  (DRB) 

The  DRB  was  established  by  the  SECDEF  in  1979  to  advise  the 
SECDEF  on  policy,  planning,  program,  and  budget  issues.  The  DRB  is  chaired 
by  the  Deputy  Secretary  of  Defense  and  is  responsible  for  the  management 
and  oversight  of  all  aspects  of  the  DoD  planning,  programming,  and  budgeting 
process.  It  oversees  the  annual  budget  review  process  and  therefore  has 
a  major  impact  on  test  and  evaluation  resources.  The  DOT&E  is  a  member 
of  the  DRB  and  can,  therefore,  have  an  impact  on  the  resources  for  T&E. 

22.3.5  Deputy  Director  Defense  Research  and  Engineering  (Test  and 

Evaluation)  (DDDRE(T&£)) 

The  DDDRE (T&E)  serves  as  the  principal  staff  assistant  and  advisor 
to  the  Director  Defense  Research  and  Engineering  for  test  and  evaluation 
matters.  He  has  authority  and  responsibility  for  all  DT&E  conducted  on 
designated  and  major  system  Research  &  Engineering,  Test  and  Evaluation 
programs.  The  DDDRE(T&E)  organization  is  illustrated  in  Figure  22-3. 

22.3.5.1  Duties  of  the  DDDRE (T&E) 

The  DDDRE(T&E)  performs  the  following  duties  within  the 
acquisition  community: 

o  Serves  as  the  focal  point  for  coordination  of  all  Test 
and  Evaluation  Master  Plans  (TEMPs).  Signs  for  approval  of  TEMPs  with 
DOT&E. 


o  Reviews  major  defense  acquisition  program  documentation 
for  DT&E  implications  and  resource  requirements,  to  provide  comments  to 
the  USD(A),  DAE  or  DAB. 
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DEFENSE  ACQUISITION  COMMITTEES 
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Figure  22-2.  Defense  Acquisition  Committee  Organization 
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Figure  22-3.  Office  of  the  Deputy  Director  Defense  Research  and  Engineering 
(Test  &  Evaluation) 
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o  Observes  DT&E  to  ensure  adequacy  of  testing  and  to  assess 
test  results. 

o  Provides  the  DAE  and  DAB  with  a  technical  assessment  of 

T&E  conducted  on  a  weapon  system. 

o  Provides  advice  and  makes  recommendations  to  the  SECDEF 
and  issues  guidance  to  the  Service  Acquisition  Executives  with  respect 
to  DT&E. 


o  Performs  the  administrative  processing  of  nominations  and 

charters  for  joint  development  test  programs. 

o  Provides  oversight  of  the  Major  Range  and  Test  Facility 

Base. 

o  Administers  the  Foreign  Weapons  Evaluation  Program  and 

NATO  Comparative  Test  Program. 

o  Confirms,  with  the  advice  from  the  Assistant  to  the  Secretary 
of  Defense  (Atomic  Energy),  that  nuclear  survivability  and  hardness 
objectives  have  been  addressed  during  DT&E. 

22.3.5.2  DDDRE(TSE)  and  Service  Reports 

The  DDDRE(T&E)  and  Services  interaction  during  the  testing  of 

major  and  designated  weapon  systems  includes  the  following  reporting 
requirements. 

o  A  TEMP  (either  initial  or  updated,  as  appropriate)  must 
be  provided  for  consideration  and  approval  15  days  before  each  milestone 
review. 


o  A  significant  T&E  Event  report  must  be  provided  to  the 
DDDRE(T&E)  within  24  hours  of  the  test  event. 

o  An  End-of-Test  Phase  Report  must  be  provided  to  DDDRE(T&E) 
listing  the  T&E  results,  conclusions,  and  recommendations  at  least  45 
days  prior  to  a  milestone  decision  or  the  final  decision  to  proceed  beyond 
LRIP. 


22.3.6  Director  Operational  Test  and  Evaluation  (DOT&E) 

As  illustrated  in  Figure  22-4,  the  Director  reports  directly 
to  the  Secretary  of  Defense  and  has  special  reporting  requirements  to 
the  Congress.  The  DOT&E 's  responsibility  to  Congress  is  to  provide  an 
unbiased  window  of  insight  into  the  operational  effectiveness  and 
suitability  of  new  weapon  systems. 
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Figure  22-4.  The  DOT&E  Organizational  Reporting  Structure 


22.3.6.1  Duties  and  Functions  of  the  DOT&E 

The  specific  duties  of  DOT&E  are  outlined  in  DoD  Directive  5141.2. 
The  functions  of  the  office  include: 

o  Obtain  reports,  information,  advice,  and  assistance  as 

necessary  to  carry  out  assigned  functions  (DOT&E  has  access  to  all  records 
and  data  in  DoD  on  acquisition  programs). 

o  Signs  the  TEMPs  for  approval  of  OT&E  and  approves  the  OT&E 
funding  for  major  systems  acquisition. 

o  Approve  test  plans  on  all  major  systems  prior  to  system 
starting  operational  test.  (Approval  in  writing  is  required  before 
operational  testing  may  begin). 

o  Provide  observers  during  preparation  and  conduct  of  OT&E. 

o  Analyze  results  of  OT&E  conducted  for  each  major  or 

designated  defense  acquisition  program,  and  submit  a  report  to  the  SECDEF 
and  Congress  on  the  adequacy  of  the  operational  test  and  evaluation 

performed. 


o  A  final  decision  to  proceed  with  a  major  program  beyond 
low-rate  initial  production  (LRIP)  cannot  be  made  until  DOT&E  has  reported 
(LRIP  report)  to  the  SECDEF  and  to  Congressional  Committees  on  Armed 
Services  and  Appropriations  on  the  adequacy  of  test  and  evaluation,  and 
whether  the  results  confirm  the  system's  operational  effectiveness  and 
suitability. 

22.3.6.2  DOT&E  and  Service  Interactions 

For  DoD  and  DOT&E  designated  acquisition  programs,  the  Service 
provides  the  DOT&E  the  following: 

o  A  draft  copy  of  the  Test  Plan  for  review. 

o  Significant  Test  Plan  changes. 

o  The  final  service  OT&E  report  must  be  submitted  to  DOT&E 

at  least  45  days  prior  to  the  DAB  Milestone  III  review. 

o  A  briefing  on  the  report  and/or  independent  evaluation 
of  the  systems. 
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SERVICE  T&E  MANAGEMENT  STRUCTURES 


22.4.1  Army  T&E  Organizational  Relationship 

The  Army  management  structure  for  test  and  evaluation  is 
illustrated  in  Figure  22-5. 

22.4.1.1  Army  Acquisition  Executive 

The  Undersecretary  of  the  Army  is  the  Army  Acquisition  Executive 
(AAE).  The  AAE  is  responsible  for  all  acquisition  T&E  (operational  and 
developmental  tests)  planning,  programming,  budgeting,  and  developmental 
tests  policy  and  oversight.  The  AAE  performs  these  duties  with  the 
assistance  of  the  Assistant  Secretary  of  the  Army,  Research,  Development, 
and  Acquisition  (ASA/RDA).  As  illustrated  in  Figure  22-5,  the  ASA/RDA 
is  organized  to  provide  technical  assessments  and  program  evaluations. 
He  resolves  acquisition  issues  whenever  possible  and  makes  recommendations 
to  the  AAE  on  the  acquisition  of  weapon  systems.  The  Deputy  Undersecretary 
of  the  Army  for  Operations  Research  (DUSA(OR))  is  chartered  to  supervise 
all  Army  T&E  policy  and  has  oversight  for  all  Army  T&E. 

22.4.1.2  Army  Technical  Testers  and  Evaluators 

o  The  U.S.  Army  Materiel  Command  (AMC)  is  responsible  for 

the  management  of  development  test  and  evaluation.  The  Test  and  Evaluation 
Command  (TECOM)  has  the  primary  responsibility  for  conducting  technical 
tests  for  the  Army  and  under  certain  conditions  conducting  the  evaluation. 
The  TECOM  is  responsible  for: 

o  Planning,  executing  and  reporting  the  results  of  technical 

tests.  Technical  tests  include  Development  Tests,  Technical  Feasibility 
Tests,  Production  Qualification  Tests,  Joint  Tests,  and  contractor/foreign 
tests. 

o  Providing  test  facilities  and  technical  expertise  in  support 
of  the  T&E  life  cycle. 

o  Maintaining  the  Army's  Major  Range  and  Test  Facility  Base. 

o  Maintaining  the  Army's  T&E  data  base. 

o  Researching,  developing,  and  acquiring  instrumentation 

and  developing  new  and  improved  test  methodology. 

o  Providing  safety  confirmations. 
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22.4.1.3  Arqy  Operational  Test  and  Evaluation 

o  The  Army  Operational  Test  and  Evaluation  Agency  (OTEA) 
is  responsible  for  the  management  of  operational  testing  of  all  major 
and  selected  nonmajor  systems  as  well  as  the  management  of  joint  user 
testing.  OTEA  is  an  independent  agency  reporting  directly  to  the  Army 
Vice  Chief  of  Staff. 

o  The  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  supports 
operational  testing  by  conducting  tests  on  selected  nonmajor  systems. 
Operational  tests  are  conducted  with  doctrine,  tactics,  and  logistic  support 
concepts  developed  by  TRADOC. 

o  The  U.S  Army  Forces  Command  (FORSCOM)  supports  testing 
by  providing  user  troops  and  facilities  as  needed. 

22.4.2  Navy  T&E  Organizational  Relationship 

The  organizational  structure  for  test  and  evaluation  in  the 
Navy  is  illustrated  in  Figure  22-6.  Within  the  Navy  Secretariat,  the 
Secretary  of  the  Navy  has  assigned  general  and  specific  RDT&E 
responsibilities  to  the  Assistant  Secretary  of  the  Navy  (Research, 
Engineering,  and  Systems)  and  to  the  Chief  Naval  Operations.  The  CNO 
has  responsibility  for  ensuring  the  adequacy  of  the  Navy's  overall  test 
and  evaluation  program.  The  T&E  policy  and  guidance  are  exercised  through 
the  Director  R&D  Requirements  and  T&E  (OP-098)  staff  support  is  provided 
by  the  Test  and  Evaluation  Division  (OP-983)  who  has  cognizance  over 
planning,  conducting  and  reporting  all  T&E  associated  with  development 
of  systems. 

22.4.2.2  Navy  DT&E  Organizations 

The  Navy's  senior  systems  development  authority  is  divided  among 
the  Commanders  of  the  System  Commands  with  NAVA I R  developing  and  performing 
DT&E  on  aircraft,  NAVSEA  developing  and  performing  DT&E  on  ships,  and 
SPAWAR  developing  and  performing  DT&E  on  all  other  systems.  System 
acquisition  is  controlled  by  a  chartered  program  manager  or  by  the  commander 
of  a  systems  command.  In  both  cases,  the  designated  developing  agency 
is  responsible  for  DT&E  and  for  the  coordination  of  all  test  and  evaluation 
planning  in  the  TEMP.  Developing  Agencies  (DA)  are  responsible  for  the 
following  testing  activities: 

o  Developing  test  issues  based  on  the  thresholds  established 
by  the  OPNAV  in  the  Operational  Requirement  or  Navy  Decision  Coordination 
paper. 


o  Identifying  the  testing  facilities  and  resources  required 
to  conduct  the  DT&E. 
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o  Developing  the  DT&E  test  reports  and  quick-look  reports. 

22.4.2.3  Navy  Operational  Test  and  Evaluation  Force 

The  Commander  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 
commands  the  Navy's  independent  operational  test  and  evaluation  activity 
and  reports  directly  to  the  CNO.  The  functions  of  the  COMOPTEVFOR  include 
the  following: 

o  Establish  early  liaison  with  the  DA  to  ensure  an 

understanding  of  the  test  requirements  and  plans. 

o  Review  acquisition  program  documentation  to  ensure  that 

documents  are  adequate  to  support  a  meaningful  T&E  program. 

o  Plan  and  conduct  realistic  OT&E. 

o  Develop  tactics  and  procedures  for  the  employment  of  systems 
that  undergo  OT&E  (as  directed  by  the  CNO). 

o  Provide  recommendations  to  the  CNO  for  the  development 

of  new  capabilities  or  the  upgrade  of  ranges. 

o  The  President  of  the  Board  of  Inspection  and  Survey 

(PRESINSURV)  also  reports  directly  to  the  CNO  and  is  responsible  for  the 
conduct  of  acceptance  trials  of  new  ships  and  aircraft  acquisitions. 

He  is  the  primary  Navy  authority  for  Production  Acceptance  Test  and 
Evaluation  of  these  systems. 

22.4.3  Air  Force  Organizational  Relationship^ 

22.4.3.1  Air  Force  Acquisition  Executive 

The  Assistant  Secretary  of  the  Air  Force  for  Acquisition  (ASAF/A) 
is  the  senior  level  authority  for  research,  development  and  acquisition 

within  the  Air  Force.  As  illustrated  in  Figure  22-7,  he  is  an  advisor 
to  the  Secretary  of  the  Air  Force,  interfaces  directly  with  the  DDDRE(T&E) 
and  DOT&E.  He  receives  both  DT&E  and  OT&E  data  and  results  as  a  part 
of  the  acquisition  decision  process.  The  ASAF/A  has  within  his  structure 
a  Military  Deputy  (Acquisition)  who  is  the  Air  Force's  primary  staff  officer 
with  responsibility  for  R&D  and  acquisition.  He  is  the  chief  advocate 
of  Air  Force  acquisition  programs,  develops  the  RDT&E  budget,  and  is 

responsible  for  establishing  Air  Force  T&E  Policy. 

22.4.3.2  Air  Force  DT&E  Organization 

The  Air  Force  Systems  Command  (AFSC)  is  the  primary  DT&E  and 
acquisition  manager.  The  AFSC  performs  all  levels  of  research,  develops 
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weapons  systems,  support  systems,  and  equipment,  and  conducts  all  DT&E. 
The  acquisition  program  managers  are  under  the  command  of  the  Conmander, 
AFSC.  Within  the  AFSC,  there  are  five  major  product  divisions 
(Aeronautical,  Armament,  Ballistic  Missiles,  Electronics,  and  Space), 
along  with  test  centers,  laboratories,  and  missile,  aircraft  and  munitions 
test  ranges. 

The  Air  Force  Logistics  Command  (AFLC)  can  also  be  a  DT&E  and 
acquisition  manager.  Once  the  weapon  system  is  fielded  and  program 
management  responsibility  has  been  transferred  from  AFSC  to  AFLC,  AFLC 
may  take  responsibility  for  developing  and  testing  system  improvements, 
enhancements,  or  upgrades. 

22.4.3.3  Air  Force  OT&E  Organizations 

The  Deputy  Chief  of  Staff,  Plans  and  Operations  is  responsible 
for  supporting  and  coordinating  the  OT&E  activities  of  the  Air  Force 
Operational  Test  and  Evaluation  Center  (AFOTEC). 

The  Commander,  Air  Force  Operational  Test  and  Evaluation  Center, 
is  responsible  to  the  Secretary  of  the  Air  Force  and  the  Chief  of  Staff 
for  the  independent  test  and  evaluation  of  all  major  and  selected  nonmajor 
systems  acquisitions.  He  is  augmented  and  supported  by  the  operational 
commands  and  others  in  planning  and  conducting  OT&E. 

The  Air  Force  Operational  Commands,  (SAC,  MAC,  TAC,  USAFE,  and 
PACAF)  develop  operational  requirements,  employment  concepts,  tactics, 
maintenance  concepts,  and  training  requirements  and  conduct  OT&E  which 
is  monitored  by  AFOTEC.  The  Operational  Commands  also  provide  operational 
concepts,  personnel,  and  resources  to  assist  AFOTEC  in  performing  OT&E 
and  coordinate  and  provide  resources  for  acquisition  programs  sponsored 
by  AFSC. 


22.4.4  Marine  Corps  Organizational  Relationship 

22.4.4.1  Marine  Corps  Acquisition  Executive 

The  Deputy  Chief  of  Staff  for  Research  and  Development  (DCS/R&D), 
Headquarters  Marine  Corps,  directs  the  total  Marine  Corps  RDT&E  effort 
to  support  the  acquisition  of  new  systems.  His  position  within  the  General 
Staff  is  analogous  to  that  of  the  Director,  RDT&E/0P-098  in  the  Navy 
structure.  The  DCS/R&D  also  reports  directly  to  the  ASN/RE&S  in  the  Navy 
Secretariat.  Figure  22-8,  illustrates  the  Marine  Corps  organization  for 
test  and  evaluation  management. 

22.4.4.2  Marine  Corps  DT&E  Organizations 

The  Commanding  General  Marine  Corps  Research,  Development  and 
Acquisition  Center  (CG  MCRD&AC)  is  the  Marine  Corps  materiel  developing 
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agent  and  has  direct  interface  with  the  Navy  Systems  Commands.  The 

CG  MCRD&AC  implements  policies,  procedures,  and  requirements  for  DT&E 
of  all  systems  acquired  by  the  Marine  Corps.  The  Marine  Corps  also  uses 
DT&E  and  OT&E  performed  by  other  Services  which  may  develop  systems  of 
interest  to  the  Corps. 

22.4.4.3  Narine  Corps  Operational  Test  and  Evaluation  Agency 

The  Marine  Corps  Operational  Test  and  Evaluation  Agency  (MCOTEA) 
is  the  independent  OT&E  activity  maintained  by  the  Marine  Corps.  Its 

function  is  analogous  to  that  performed  by  OPTEVFOR  in  the  Navy.  The 
CG  MCRD&AC  provides  direct  assistance  to  MCOTEA  in  the  planning,  conduct 
and  reporting  of  OT&E.  The  Fleet  Marine  Force  performs  troop  test  and 

evaluation  of  materiel  development  in  an  operational  environment. 

22.5  SUtWARY 

An  increased  emphasis  on  test  and  evaluation  has  placed  greater 
demands  on  the  OSD  and  DoD  Components  to  carefully  structure  organizations 
and  resources  to  ensure  maximum  effectiveness.  Renewed  interest  by  the 

Congress  on  testing  as  a  means  of  assessing  systems  utility  and 
effectiveness  and  a  recent  report  by  the  President's  Blue  Ribbon  Panel 
on  Acquisition  Management  have  resulted  in  major  reorganizations  within 
the  Services.  These  reorganizations  will  be  ongoing  for  several  years 
to  improve  the  program  management  and  test  and  evaluation  of  acquisition 
systems. 
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CHAPTER  23 

PROGRAM  OFFICE  RESPONSIBILITIES  FOR  TEST  AND  EVALUATION 


23.1  INTRODUCTION 

In  Government  Program  Management  Offices  (PMO),  there  should 
be  an  element  dedicated  to  management  of  Test  and  Evaluation.  This  element 
would  have  the  overall  test  program  responsibility  for  all  phases  of 
the  acquisition  process.  In  the  PMO,  the  Deputy  for  Test  and  Evaluation 
( T&E )  would  be  responsible  for  defining  the  scope  and  concept  of  the 
test  program,  establishing  the  overall  program  test  objectives  and  managing 
test  program  funds  and  coordination.  The  Deputy  for  T&E  should  provide 
test  directors  as  required,  such  as  a  Joint  Test  Director,  and  coordinate 
the  test  resources,  facilities,  and  their  support  required  for  each  phase 
of  testing.  In  addition,  he  or  a  member  of  his  staff,  will  be  responsible 
for  managing  the  Test  and  Evaluation  Master  Plan  (TEMP)  and  planning 
and  managing  the  special  test  programs  required  for  the  program.  The 
Deputy  (T&E)  will  also  review,  evaluate,  approve,  and  release  for 
distribution,  contractor  prepared  test  plans  and  reports,  and  review 
and  coordinate  all  appropriate  government  test  plans.  Afte*  the  system 
is  produced,  he  will  be  responsible  for  supporting  Product  on  Acceptance 
Testing  and  the  test  portions  of  P3I  upgrades  or  enhancements  to  the 
weapon  syjtem/acquisition.  If  the  program  is  large  enough.  Deputy  (T&E) 
will  be  responsible  for  all  T&E  direction  and  guidance  on  that  program. 

23.2  RELATIONSHIP  TO  THE  PROGRAM  MANAGER 

The  Program  Manager  (PM)  is  ultimately  responsible  for  all 
aspects  of  the  system  development,  to  include  testing.  The  Deputy  (T&E) 
is  normally  authorized  by  the  Program  Manager  to  conduct  all  duties  in 
the  area  of  test  and  evaluation.  The  Deputy  (T&E)  input  to  the  contract, 
engineering  specifications,  budget,  program  schedule,  etc.  is  essential 
for  the  Program  Manager  to  efficiently  manage  the  program. 

23.3  EARLY  PROGRAM  STAGES 

In  the  early  stages  of  the  program,  the  Deputy  (T&E)  writes 
the  test  sections  of  the  Request  for  Proposal  (RFP).  Although  the  ultimate 
responsibility  for  the  RFP  is  between  the  Program  Manager  (PM)  and  the 
Primary  Contracting  Officer  (PCO),  the  Deputy  (T&E)  has  the  responsibility 
for  creation  of  several  sections.  These  sections  include  the  test 
schedule,  test  program  funding  (projections),  test  data  requirements 
for  the  program  (test  reports,  plans,  procedures,  quick-look  reports, 
etc.),  the  test  section  of  the  Statement  of  Work  (SOW),  the  Acquisition 
Plan,  Information  for  Proposal  Preparation  (IFPP),  and  if  a  joint 
acquisition  program,  the  Joint  Requirements  Document  (JRD). 
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23.3.1  Memorandums 


Another  task  of  the  Deputy  (T&E )  early  in  the  program  is  the 
arrangement  of  any  Memorandums  of  Agreement  or  Understanding  (MOA/MOU) 
between  either  Services,  NATO  countries.  Test  Organizations,  etc.  which 
outline  the  responsibilities  of  each  organization.  The  RFP  outlines 
contractor/Government  obligations  and  arrangements  on  the  access  and 
use  of  test  facilities  (both  contractor  and  Government  owned). 

23.3.2  Test  Data  Management 

The  Deputy  (T&E)  may  have  approval  authority  for  all  contractor 
created  test  plans,  procedures,  and  reports.  He  must  have  access  to 
all  contractor  testing  and  test  results  and  he  is  responsible  for 
disseminating  the  results  to  Government  agencies  that  need  this  data. 
Additionally,  the  Deputy  creates  report  formats  and  time  lines  for 
contractor  submittal,  government  approval,  etc. 

The  data  requirements  for  the  entire  test  program  are  outlined 
in  the  Contract  Data  Requirements  List  (CDRL).  The  Deputy  (T&E)  provides 
input  to  this  section  of  the  RFP  early  in  the  program.  He  ensures  that 
his  office  and  all  associated  test  organizations  requiring  the  information 
are  distributed  the  test  documentation  in  a  timely  manner.  Usually, 
the  contractor  sends  the  data  packages  directly  to  the  Deputy  (T&E)  who, 
in  turn,  has  a  distribution  list  trimmed  down  to  the  minimum  amount  of 
copies  for  agencies  needing  that  information  to  perform  their  mission 
and  oversight  responsibilities.  It  is  important  for  the  Deputy  (T&E) 
to  use  an  integrated  test  program  and  request  contractor  test  plans  and 
procedures  well  in  advance  of  the  actual  performance  of  the  tests  to 
ensure  his  office  has  time  to  approve  the  procedures  and  effect  corrections 
or  modifications.  Conversely,  he  must  also  receive  the  test  results 
and  reports  in  a  timely  manner  to  enable  him,  the  program  manager,  and 
higher  authorities  to  make  program  decisions.  Further,  the  data  received 
should  be  tailored  to  provide  the  minimum  information  and  copies  needed. 
The  Deputy  (T&E)  must  remain  aware  of  the  fact  that  data  requirements 
in  excess  of  the  minimum  needed  will  lead  to  an  unacceptable  increase 
in  overall  program  cost.  For  data  that  is  needed  quickly  and  informally 
(at  least  initially),  the  Deputy  (T&E)  can  request  Quick-Look  Reports 
that  give  test  results  immediately  after  test  performance.  The  Deputy 
(T&E)  is  also  responsible  for  coordinating  with  the  contractor  on  all 
report  formats  (usually  the  in-house  contractor  format  is  acceptable 
in  most  cases). 

23.3.3  Test  Schedule  Formulation 

A  very  important  task  the  Deputy  (T&E)  has  for  the  creation 
of  the  RFP  is  the  test  program  schedule.  Initially,  the  program  manager 
will  need  contractor  predictions  of  the  hardware  (and  software  in  some 
cases) 
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availability  dates  for  models,  prototypes,  mockups,  full-scale  models, 
etc.  once  the  contract  is  awarded.  The  Deputy  (T&E)  uses  this  information 
to  create  a  realistic  front-end  schedule  of  the  in-house  testing  the 
contractor  will  conduct  prior  to  government  testing  (DT  &  OT).  Then, 
a  "strawman"  schedule  is  developed  upon  which  the  government  DT  and  OT 
schedules  can  be  formulated  and  contractor  support  requirements  determined. 
The  Deputy  ( T&E )  can  use  past  experience  in  testing  similar  weapon 

systems/acquisition  items  or  contact  test  organizations  which  have  the 
required  experience  to  complete  the  entire  test  schedule.  Since  the 
test  schedule  is  a  critical  contractual  item,  the  contractor's  inputs 

are  very  important.  The  test  schedule  will  normally  become  an  item  for 
negotiation  once  the  RFP  is  released  and  the  contractor's  proposal 
received.  Attention  must  be  given  to  ensuring  the  test  schedule  is  not 
so  "success  oriented"  in  such  a  way  that  any  test  failures  will  result 
in  serious  consequences  for  either  the  government  test  agencies  or  the 
contractor. 

23.4  PMO/CONTRACTOR  TEST  MANAGEMENT 

The  PMO  will,  in  most  cases,  have  a  contractor  test  section 

counterpart.  With  this  counterpart,  the  Deputy  (T&E)  works  out  the 
detailed  test  planning,  creation  of  schedules,  etc.  for  the  entire  test 
program.  The  PMO  uses  inputs  from  all  sources  (contracts.  Development 
Test  Agencies,  Operational  Test  Agencies,  Higher  Headquarters,  etc.) 
to  formulate  the  test  program's  length,  scope,  and  necessary  details. 

The  Deputy  (T&E)  ensures  that  the  RFP  reflects  the  precise  test  program 
envisioned  and  the  contractor's  role  in  the  acquisition.  He  also  ensures 
that  the  RFP  includes  provisions  for  Government  attendance  at  contractor's 
tests  and  that  all  contractor  test  results  are  provided  to  the  Government. 

Once  the  RFP  has  been  submitted  and  the  contractor's  proposal 
is  received,  it  is  reviewed  by  the  PMO.  The  Deputy  (T&E)  is  responsible 
for  performing  a  technical  evaluation  on  the  test  portions  of  the  proposal. 
In  this  technical  evaluation,  he  compares  the  proposal  to  the  Statement 
of  Work,  test  schedule,  IFPP,  etc.  and  reviews  the  contractor's  costing 
of  each  testing  item.  This  is  an  iterative  process  of  refining, 
clarifying,  and  modifying  that  will  ensure  the  final  contract  between 
the  PMO  and  the  Prime  Contractor  (Subcontractors)  contains  all  test  related 
tasks  and  is  priced  within  scope  of  the  proposed  test  program.  Once 
technical  agreement  on  the  contractor's  technical  approach  is  reached, 
the  Deputy  (T&E)  is  responsible  for  giving  inputs  to  the  government 
contracting  officer  during  contract  negotiations.  The  contracting  officer 
requests  contract  deliverables  to  which  are  assigned  contract  line  item 
numbers  (CLIN)  which  are  created  by  the  Deputy  (T&E).  This  will  ensure 
the  Contractor  delivers  the  required  performances  at  specified  intervals 
during  the  life  of  the  contract.  Usually,  there  will  be  separate  contracts 
for  development  and  production  of  the  acquisition  item.  For  each  type 
of  contract,  the  Deputy  (T&E)  has  the  responsibility  to  provide  the  PCO 
and  PM  with  the  test  and  evaluation  inputs  to  each. 
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23.5  TEST  PLANNING  WORKING  GROUPS 

Prior  to  the  creation  of  the  final  version  of  the  RFP,  the 
Deputy  (T&E )  will  form  a  test  planning  working  group.  This  group  includes 
the  Operational  Test  Agency,  Development  Test  Agency,  any  organizations 
that  may  be  jointly  acquiring  the  same  system,  the  test  supporting 
agencies,  operational  users,  and  any  other  organizations  that  will  be 
involved  in  the  test  program  by  providing  test  support,  conducting, 
evaluating,  or  reporting  on  testing.  In  most  cases,  the  contractor 
participates  in  this  test  planning  group;  however,  the  contractor  may 
not  be  selected  by  the  time  the  first  meetings  are  held. 

The  purpose  of  these  meetings  are  to  review  and  assist  in  the 
development  of  the  Test  and  Evaluation  Master  Plan  (TEMP)  and  to  reach 
agreement  on  basic  test  program  schedules,  scope,  support,  etc.  The 
TEMP  serves  as  the  top  level  test  management  document  for  the  acquisition 
program,  changing  and  being  updated  as  the  test  program  dictates  in  the 
future. 

23.6  TEST  PROGRAM  FUNDING/BUDGETING 

The  PMO  must  identify  funds  for  testing  very  early  so  that 
test  resources  can  be  obtained.  The  Deputy  (T&E)  uses  the  acquisition 
schedule,  TEMP  and  other  program  and  test  documentation  to  identify  test 
resource  requirements.  He  coordinates  these  requirements  with  the 
Government  organizations  that  have  the  test  facilities  to  ensure  their 
availability  for  testing.  The  Deputy  T&E  ensures  that  test  costs  include 
both  the  contractor  and  the  government  test  costs.  The  contractor's 
test  costs  are  normally  adequately  outlined  in  his  proposal,  whereas 
the  Government  test  ranges,  instrumentation,  and  test  support  resource 
costs  must  be  determined  by  other  means.  Usually,  the  Deputy  (T&E) 
contacts  the  test  organization,  outlines  his  test  program  requirements 
and  the  test  organization  sends  the  program  office  an  estimate  of  the 
test  program  costs.  He  then  obtains  cost  estimates  from  all  test  sources 
he  anticipates  using  and  supplies  this  information  to  the  Program  Manager. 
The  Deputy  (T&E)  must  also  ensure  that  any  funding  reductions  on  the 
program  are  not  absorbed  entirely  by  the  test  program.  Some  cutbacks 
may  be  necessary  and  allowable,  but  the  test  program  must  supply  him, 
other  defense  decision  making  authorities,  and  Congress  with  enough 
information  to  make  program  milestone  decisions. 

23.7  TECHNICAL  REVIEWS,  DESIGN  REVIEWS,  AND  AUDITS 

The  role  of  the  Deputy  (T&E)  changes  slightly  during  the 
contractor's  technical  reviews,  design  reviews,  physical  and  functional 
configuration  audits,  etc.  Whereas,  usually  he  plans,  directs,  or  monitors 
government  testing,  in  the  reviews  and  audits,  he  examines  the  contractor's 
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approach  to  the  test  problem  and  evaluates  the  validity  of  the  process 
and  the  accuracy  of  the  contractor's  results.  Using  his  experience  and 
background  in  test  and  evaluation,  he  also  assesses  whether  the  contractor 
did  enough  or  too  little  testing,  whether  the  tests  were  biased  in  any 
way,  and  if  they  followed  a  logical  progression  using  the  minimum  of 
time,  effort,  and  funds.  If  the  Deputy  (T&E)  finds  any  discrepancies, 
he  must  inform  the  contractor,  the  program  manager,  and  the  primary 
contracting  officer  to  validate  his  conclusions  and  then  effect 
corrections.  Each  type  of  review  or  audit  will  have  a  different 
focus/orientation,  but  the  Deputy  (T&E)  will  always  be  concerned  with 
the  testing  process  and  how  it  was  carried  out.  After  each  review,  the 
Deputy  (T&E)  should  always  document  his  observations  for  future  reference. 

23.8  CONTRACTOR  TESTING 

The  Deputy  (T&E)  is  responsible  for  monitoring  all  contractor 
conducted  tests.  He  must  also  be  given  access  to  all  contractor  internal 
data,  test  results,  and  test  reports  related  to  his  acquisition  program. 
Usually,  the  contract  outlines  the  requirement  that  the  government 
representatives  be  informed  ahead  of  time  of  any  (significant  or  otherwise) 
testing  the  contractor  conducts  so  the  government  can  arrange  to  witness 
the  testing  or  receive  results  of  the  tests.  Further,  the  contractor's 
internal  data  should  be  available  as  a  contract  provision.  The  Deputy 
(T&E)  must  ensure  that  Government  test  personnel  (DT&E/OT&E)  have  access 
to  contractor  test  results.  It  would  be  desirable  to  have  all  testers 
observe  the  contractor  tests  to  help  develop  confidence  in  the  system 
and  identify  areas  of  risk. 

23.9  SPECIFICATIONS 

Within  the  program  office,  the  Engineering  Section  is  usually 
tasked  to  create  the  preliminary  specifications  for  release  of  the  RFP. 
The  contractor  is  then  tasked  with  creation  of  the  specifications  in 
the  contract,  which  will  be  delivered  once  the  item/system  design  is 
formalized  for  production.  The  Deputy  (T&E)  becomes  involved  in 

specification  formulation  in  an  important  way.  He  reviews  the 

specifications  with  an  insight  to  determine  if  they  are  testable,  if 

current  technology  or  state-of-the-art  technical  means  can  determine 
(during  the  DT&E  test  phase)  if  the  spec's  are  being  met  by  the  acquisition 
item,  or  if  the  specification  is  too  "tight".  A  specification  is  too 
"tight"  if  the  requirements  are  impossible  to  meet  or  test  towards,  or 
the  specification  has  no  impact  on  appearance  or  performance  of  the  end 
item,  the  system  it  will  become  a  part  of  or  the  system  it  will  interact 
with.  He  must  determine  if  test  objectives  can  be  adequately  formulated 
from  those  spec's  at  later  dates  that  will  provide  thresholds  of 
performance,  minimum  and  maximum  standards,  and  reasonable  operating 
conditions  for  the  end  item's  final  purpose  and  operating  environment. 
The  specifications  shape  the  DT&E  testing  scenario,  test  ranges,  test 

support,  targets,  etc.  and  so  are  very  important  to  the  Deputy  (T&E). 
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23.10  INDEPENDENT  EVALUATION  AGENCIES 

The  PMO  Deputy  (T&E)  does  not  have  direct  control  over 
government-owned  test  resources,  test  facilities,  test  ranges,  test 
personnel,  etc.  Therefore,  he  must  depend  on  those  test  organizations 
controlling  them.  However,  the  Deputy  (T&E)  must  stay  involved  with 
the  test  agency  activities.  The  amount  of  involvement  depends  on  the 
item  being  tested,  its  complexity,  cost,  characteristics,  the  length 
of  time  for  testing,  amount  of  test  funds,  etc.  Usually,  the  "nuts  & 
bolts"  detailed  test  plans  and  procedures  are  written  by  the  test 
organizations  controlling  the  test  resources  with  inputs  and  guidance 
from  the  Program  Office  (Deputy  (T&E)).  The  Deputy  (T&E)'s  responsibility 
is  to  ensure  that  the  tests  are  performed  using  test  objectives  based 
upon  the  specifications  and  that  the  requirements  of  timeliness,  accuracy, 
and  minimal  costs  are  met  by  the  test  program  design.  During  the  testing, 
the  Deputy  (T&E)  monitors  the  testing.  The  test  agencies  submit  their 
a  copy  of  their  report  to  the  Program  Office  at  the  end  of  testing,  usually 
to  the  Office  of  the  Deputy  (T&E). 
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CHAPTER  24 

PROGRAM  MANAGEMENT  OPERATIONAL  TEST  RESPONSIBILITIES 

24.1  INTRODUCTION 

In  the  Government  Program  Management  Office  (PMO),  there  is 
a  section  dedicated  to  Test  and  Evaluation  (T&E).  Besides  being 
responsible  for  Development  Test  &  Evaluation  (DT&E)  support  to  the  Program 
Manager,  this  section  is  also  responsible  for  the  program  coordination 
of  Operational  Test  and  Evaluation  (OT&E).  The  office  of  the  Deputy 

for  Test  and  Evaluation  (Deputy  (T&E))  is  designated  to  provide  this 

support  to  the  Program  Manager.  In  some  Services,  responsibilities  of 

the  Deputy  (T&E)  include  coordination  of  test  resources  for  all  phases 

of  OT&E. 

24.2  CONTRACT  RESPONSIBILITIES 

The  Deputy  (T&E),  or  his  representative,  ensures  that  certain 

sections  of  the  Request  for  Proposal  (RFP)  contain  sufficient  allowance 
for  T&E  support  by  contractors.  This  applies  whether  the  contract  is 
for  a  development  item,  a  production  item  (limited  production,  such  as 
LRIP  or  full-rate  production)  or  the  enhancement/upgrade  of  portions 

of  a  weapons  system.  Where  allowed,  within  the  law,  contractor  support 
should  be  considered  to  help  resolve  basic  issues  such  as  data 

requirements,  test  schedules,  contractor  test  support  and  funding. 

In  the  overall  portion  of  the  RFP,  all  Government  personnel, 

especially  those  in  the  Operational  Test  Agencies,  must  be  guaranteed 

access  to  the  contractor's  development  facilities,  especially  during 
the  DT&E  phase.  The  Government  representatives  must  be  allowed  to  observe 
all  contractor  in-house  testing  and  have  access  to  his  test  data  and 

reports. 

24.2.1  Data  Requirements 

The  contract  must  specify  the  data  the  contractor  must  supply 
during  OT&E.  Unlike  DT&E,  the  contractor  will  not  be  making  the  test 
plans,  procedures,  or  reports.  These  documents  are  the  responsibility 
of  the  Operational  Test  Agency  (OTA).  The  Deputy  (T&E)  should  include 
the  OTA  on  a  distribution  list  for  all  test  documents  which  may  concern 
them  during  the  DT&E  phase  of  testing  to  keep  them  informed  on  the  test 
item's  progress  and  previous  testing.  In  this  way,  the  OTA  will  be  better 
informed  when  developing  their  own  test  plans  and  procedures  for  OT&E. 
In  fact,  the  OTA  should  attend  the  Contract  Data  Requirements  List  (CDRL) 
Review  Board  and  list  for  the  PMO  the  types  of  documents  the  OTA  will 
need.  The  Deputy  (T&E)  should  coordinate  the  test  sections  of  this  data 
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list  with  the  OTA  and  then  represent  their  concerns  in  that  meeting. 
All  tests  the  contractor  performs  should  be  reported  and  copies  of  those 
reports  made  available  to  the  OTA.  In  return,  the  Deputy  (T&E)  must 
ensure  that  he  is  kept  informed  about  all  OTA  activities  and  receive 
their  test  procedures,  test  plans,  and  their  test  reports.  Unlike  DT&E, 
the  Deputy  (T&E)  will  not  have  report  or  document  approval  authority 
as  he  does  over  the  contractor's  documentation.  The  Deputy  (T&E)  is 
always  responsible  for  keeping  the  program  manager  informed  on  OT&E 
progress. 


24.2.2  Test  Schedule 

Another  important  early  activity  the  Deputy  (T&E)  must  accomplish 
is  to  determine  the  OT&E  test  schedule.  Since  the  contractor  may  need 
to  provide  support  (depending  on  whether  the  contractor  will  maintain 
the  acquisition  item  in  the  field  once  operational),  the  test  schedule 
may  need  to  be  contractually  agreed  to  before  contract  award.  Sometimes, 
the  Deputy  (T&E)  can  formulate  a  strawman  schedule  (based  on  previous 
experience  with  like  items)  and  then  present  this  schedule  to  the 
operational  test  representative  at  the  initial  test  planning  working 
group  for  their  review.  Or,  he  can  simply  contact  the  OTA,  arrange  a 
meeting  to  discuss  the  new  program,  and  in  that  meeting  discuss  time 
requirements  that  the  OTA  envisions.  That  input  then  goes  into  the  RFP 
and  to  the  Program  Manager.  The  test  schedule  must  allow  for  time  between 
DT&E  testing  and  OT&E  testing  if  the  testing  is  not  combined  or  the  test 
assets  are  limited.  That  time  gap  is  necessary  for  review  of  DT&E  test 
results,  set-up  of  OT&E,  refurbishment  or  corrections  of  deficiencies 
discovered  during  DT&E,  etc.  The  test  schedule  for  DT&E  should  not  be 
so  "success  oriented"  that  the  OT&E  test  schedule  is  adversely  impacted, 
not  allowing  enough  time  for  adequate  testing,  or  the  reporting  of  OT&E 
results.  For  example,  if  the  DT&E  schedule  slips  6  months,  the  OT&E 
schedule  and  the  milestone  decision  should  slip  also.  In  the  event  of 
a  schedule  slip,  OT&E  should  not  be  shortened  just  to  make  a  milestone 
decision  date. 

24.2.3  Contractor  Support 

The  Deputy  (T&E),  being  responsible  for  providing  all  T&E  inputs 
to  the  RFP,  must  determine  early  in  the  program  acquisition  phase,  whether 
the  contractor  will  be  involved  in  supporting  OT&E  and,  if  so,  to  what 
extent  that  support  will  be.  According  to  Congress,  the  contractor  can 
only  be  involved  in  the  conduct  of  OT&E  if,  once  the  item  is  fielded, 
the  contractor  will  be  providing  the  support  of  that  item  or  operating 
that  item.  If  not,  no  contractor  support  is  allowed  during  OT&E.  Prior 
to  OT&E,  however,  the  contractor  may  be  tasked  with  providing  training 
and  handbooks  to  typical  operational  users  and  maintenance  personnel. 
In  addition,  the  contractor  must  be  required  to  provide  sufficient  spare 
parts  for  the  operational  maintenance  personnel  to  maintain  the  test 
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item  while  undergoing  operational  testing.  These  support  items  must 
be  agreed  to  by  the  PMO  and  OTA  and  then  made  contractually  binding  on 
the  contractor.  If,  however,  the  contractor  will  be  required  to  provide 
on-site  maintenance  of  the  item  for  the  duration  of  its  useful  life, 
then  the  contractor  will  be  allowed  (and  obligated)  to  participate  in 
OT,  to  include  spare  parts,  training,  etc. 

24.2.4  OT&E  Funding 

The  Deputy  (T&E)  helps  provide  the  Program  Manager  estimates 
of  all  test  program  costs  to  conduct  pre-production  OT&E.  This  funding 
includes  both  contractor  and  Government  test  support  for  which  the  Program 
Office  directly  or  indirectly  will  be  responsible.  Some  OTA  support 
is  funded  by  the  Program  Office  for  conducting  OT&E  on  Government  test 
ranges.  The  Deputy  (T&E)  must  determine  these  costs  and  inform  the  Program 
Manager.  The  contractor's  funding  for  DT&E  will  be  handled  by  the 
contracts  (development  and  production). 

24.2.5  Statement  of  Work 

The  most  important  document  the  Deputy  (T&E)  provides  inputs 
to  is  the  Statement  of  Work  (SOW).  In  this  document,  he  must  outline 
all  required  anticipated  contractor  support  for  DT&E  and  OT&E.  This 
document  outlines  the  data  requirements,  contractor  conducted  or  supported 
testing,  government  involvement  (access  to  the  contractor's  data,  tests, 
and  results),  operational  test  support,  and  any  other  specific  test 
requirements  the  contractor  will  be  tasked  to  perform  during  the  duration 
of  the  contract. 

24.3  TEST  AND  EVALUATION  MASTER  PUN 

The  Test  and  Evaluation  Master  Plan  (TEMP)  should  be  updated 
as  required  during  OT.  The  Deputy  (T&E)  is  responsible  for  managing 
the  TEMP  throughout  the  test  program.  The  Operational  Test  Agency  usually 
is  tasked  to  complete  the  operational  test  section  of  the  TEMP  and  provide 
Operational  Test  Plans  (OTPs)  outlining  their  proposed  test  program  through 
all  phases  of  OT&E.  It  is  important  to  keep  the  TEMP  updated  regularly 
so  that  test  organizations  involved  in  OT&E  understand  the,  scope  of  their 
test  support.  Further,  if  any  upgrades,  improvements,  or  enhancements 
to  the  fielded  weapon  system  occur,  the  TEMP  must  be  updated  or  a  new 
one  created  to  include  any  new  DT  and  OT  requirements. 

24.4  PHASES  OF  OPERATIONAL  TEST 

The  Deputy  (T&E)  performs  different  roles  during  each  phase 
of  operational  test.  The  phases  include  Initial  Operational  Test  & 
Evaluation  (IOT&E)  and  Follow-On  Operational  Test  &  Evaluation  (FOT&E). 
For  IOT&E,  the  Deputy  (T&E)  must  ensure  the  contract  portions  are  adequate 
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to  cover  the  scope  of  testing  as  outlined  by  the  Operational  Test  Agency. 
The  Program  Office  may  also  provide  a  Test  Director  to  represent  the 
Deputy  (T&E)  during  the  actual  testing.  The  Deputy  (T&E ) ' s  involvement 
in  IOT&E  will  be  more  of  a  monitoring  and  coordination  mode  wherein  he 
keeps  the  Program  Manager  informed  of  progress  and  problems  that  arise 
through  testing  and  provides  whatever  support  to  the  test  organization 
that  is  required.  For  any  problems  requiring  Program  Office  support, 
the  Deputy  (T&E)  will  be  the  point  of  contact. 

During  IOT&E,  the  Deputy  (T&E)'s  responsibility  is  to  ensure 
the  contract  allows  adequate  insight  into  contractor  production  testing, 
to  Include  qualification  testing.  Further,  the  production  contract  has 
to  outline  contractor  support  of  operational  testing,  to  include  the 
production  of  adequate  spare  parts  the  Service's  maintenance  personnel 
will  need  to  maintain  the  acquisition  item/weapon  system  during  OT&E. 
Also,  enough  Low-Rate  Initial  Production  (LRIP)  items  must  be  manufactured 
to  run  a  complete  and  adequate  OT&E  program.  If  the  contractor  will 
maintain  the  item  in  the  field,  then  the  contractor  must  be  a  part  of 
the  IOT&E. 

During  FOT&E,  the  Deputy  (T&E)  monitors  the  testing  and  usually 
no  contractor  or  contract  is  involved  with  this  phase.  If  inadequacies 

are  noted  during  FOT&E,  the  program  manager  and  the  engineering  section 

of  the  Program  Office  may  design  or  develop  modifications,  which  may 
be  Incorporated  into  the  weapon  system  design.  The  Deputy  (T&E)  should 
receive  any  reports  generated  by  the  operational  testers  during  this 
time.  Any  deficiencies  noted  during  FOT&E  should  be  reported  to  the 
PMO  which  may  decide  to  incorporate  upgrades,  enhancements,  or  additions 
to  the  current  system. 

24.5  UPGRADES,  ENHANCEMENTS,  ADDITIONS 

Once  a  weapon  system  is  fielded,  portions  of  that  system  may 

become  obsolete,  ineffective,  or  deficient  and  need  replacement,  upgrading, 
or  enhancing  to  ensure  the  weapon  system  can  meet  current  and  future 

requirements.  The  Deputy  (T&E)  plays  a  vital  role  in  this  process. 
The  modifications  to  existing  weapon  systems  must  be  managed,  as  would 
entire  newly  acquired  weapon  systems.  However,  since  these  are  changes 
to  existing  systems,  the  Deputy  (T&E)  has  the  responsibility  to  determine 
if  these  enhancements  degrade  the  existing  system,  are  compatible  with 
its  Interfaces  and  functions,  and  whether  the  Nondevelopment  Items  (NDIs) 
require  retest,  or  the  entire  weapon  system  needs  re- verification.  The 
Deputy  (T&E)  must  plan  the  test  program's  funding,  schedule,  test  program, 
and  contract  provisions  with  these  items  in  mind.  A  new  TEMP  may  have 
to  be  generated  or  the  original  weapon  system  TEMP  modified  and 
re-coordinated  with  the  test  organizations.  The  design  of  the  test  program 
usually  requires  coordination  with  the  engineering,  contracting,  and 
program  management  sections  of  the  Program  Office. 
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POST-PRODUCTION  DECISION  ACTIVITIES 


The  Deputy  (T&E)  will  be  Involved  with  OT&E  of  the  actual 
production  units  after  a  limited  number  are  produced.  The  IOT&E  that 
occurs  at  that  time  must  be  closely  monitored  so  that  a  full-rate 
production  decision  can  be  made.  As  in  the  Operational  Assessments, 
the  Deputy  ( T&E )  will  be  monitoring  test  procedures  and  results  and  keep 
the  Program  Manager  informed.  If  the  item  does  not  succeed  during  IOT&E, 
a  new  process  of  DT&E,  or  modification,  may  result  in  which  the  Deputy 
(T&E)  will  be  involved  (as  in  any  new  programs  inception).  If  the  item 
passes  IOT&E  testing  and  is  produced  at  full  rate,  the  Deputy  (T&E)  will 
be  responsible  for  ensuring  that  testing  of  those  production  items  is 
adequate  to  ensure  that  those  end  items  physically  and  functionally 
resemble  the  development  items. 

24.7  INDEPENDENT  EVALUATION  AGENCIES 

During  the  IOT&E,  the  Service  Operational  Test  Agency  (OTA) 

controls  all  aspects  of  testing,  to  include  test  plans,  procedures, 
reports,  etc.  The  OTA  uses  the  resources  of  other  organizations  to  test 
the  item  in  an  operational  environment  using  as  many  actual  end  item 
operators  as  possible.  The  Deputy  (T&E)'s  role  for  IOT&E  testing  includes 
ensuring  enough  funds  are  projected  for  operational  testing,  assisting 
the  OTA  in  the  coordination  of  test  resources  (including  contractor 
support)  for  OT,  monitoring  OT&E  and  providing  other  test  support  as 
required  by  the  OTA.  The  OTA  will  make  their  own  independent  evaluation 
of  OT  and  forward  copies  to  the  Program  Office  after  testing  is  complete. 
The  results  of  testing,  however,  are  usually  required  long  before  the 
report  is  finished  for  the  production  decision. 

24.8  TEST  RESOURCES 

During  all  phases  of  OT,  the  Deputy  (T&E)  must  be  concerned 

with  ensuring  the  operational  testers  have  the  test  resources  they  need 
to  accomplish  their  mission.  Test  resources  will  be  either  contractor 
owned  or  Government  owned.  The  contractor  resources  must  be  covered 
in  the  contract,  whether  in  the  development  contract  (IOT&E)  or  the 
production  contract  (FOT&E).  The  Government  test  resources  used  are 
determined  by  the  operational  testers.  They  usually  coordinate  the  test 
ranges,  test  support,  and  the  personnel  for  testing.  The  program  manager 
identifies  funding  for  his  support  of  OT.  The  funds  are  released  to 
the  OTA  to  use  for  their  test  program.  The  OTA  then  makes  a  budget  and 

obligates  funds  for  test  ranges,  instrumentation,  etc.  according  to  their 

operational  test  plans. 
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CHAPTER  25 

T&E  OF  WEAPON  SYSTEMS  TYPES 


25.1  INTRODUCTION 

This  chapter  will  offer  guidance  to  Department  of  Defense 
personnel  whose  task  it  is  to  plan,  monitor,  and  execute  test  and 
evaluation.  Checklists  for  the  chapter  were  obtained  from  the  Defense 
Science  Board  Study,  entitled:  Report  of  Task  Force  on  Test  and  Evaluation, 
dated  April  2,  1974.  This  excellent  study  is  highly  regarded  in  the 

Test  &  Evaluation  (T&E)  community.  It  has  become  dated  and  the  Defense 
Systems  Management  College  decided  to  update  the  study  findings  and  include 
those  findings  and  summary  checklists  in  this  management  guide. 

25.2  General  Test  and  Evaluation  Issues 

The  Defense  Science  Board  (DSB)  report  presented  guidance  on 
T&E  at  two  levels.  At  the  more  general  level,  it  discussed  a  number 
of  general  issues  which  were  appropriate  to  all  weapon  acquisition 
programs.  These  issues,  along  with  a  summary  discussion  are  given  below. 

25.2.1  Effects  of  Test  Requirements  on  System  Acquisition 

The  acquisition  strategy  for  the  system  should  allow  sufficient 
time  between  the  end  of  demonstration  testing  and  procurement,  as 

contracted  with  limited  production  decisions,  to  allow  flexibility  for 
modification  of  plans  which  will  be  required;  ensure  that  sufficient 

dollars  are  available  not  only  to  conduct  T&E  but  to  allow  for  the 

additional  T&E  which  is  always  required  due  to  failure,  design  changes, 
etc.;  and,  be  evaluated  relative  to  constraints  imposed  by: 

o  The  level  of  system  testing  at  various  stages  of  the  RDT&E 

cycle, 

o  The  number  of  test  items  available  and  the  schedule  interface 

with  other  systems  needed  in  the  tests,  such  as  aircraft, 
electronics,  etc. 

o  The  support  required  to  assist  in  the  preparation,  conduct 

of  the  tests,  and  the  analysis  of  the  test  results; 

o  Be  evaluated  to  minimize  the  so-called  T&E  gap  caused  by  lack 

of  hardware  during  the  test  phase. 
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25.2.2  Test  Requirements  and  Restrictions 

Tests  should: 

o  Have  specific  objectives; 

o  List  in  advance  actions  to  be  taken  as  a  consequence  of  the 
test  results; 

o  Be  instrumented  to  permit  diagnosis  of  the  cause  of  lack  of 
performance  including  random,  design  induced,  wearout,  and 
operator  error  failure;  and 

o  Not  be  repeated  if  failures  occur,  without  a  detailed  analysis 
of  the  failure.  ("Most  likely  the  failure  will  not  go  away.") 

25.2.3  Trouble  Indicators 

Establish  an  early  detection  scheme  to  identify  program  illness. 

When  a  program  begins  to  have  trouble  there  are  indicators  which 
will  show  up  during  testing.  Some  of  these  indicators  are: 

o  A  test  failure; 

o  Any  repetitive  failure; 

o  A  revision  of  schedule  or  incremental  funding  that  exceeds 
the  original  plan;  or 

o  Any  relaxation  of  the  basic  requirements  such  as  lower 
performance. 

25.2.4  Requirement  For  Test  Rehearsals 

Test  rehearsals  should  be  conducted  for  each  new  phase  of 

testing. 

25.3  SCHEDULING 

Specific  issues  associated  with  test  scheduling  are  listed 

below. 

25.3.1  Building  Block  Test  Scheduling 

The  design  of  a  set  of  tests  to  demonstrate  feasibility  prior 

to  the  Full-Scale  Development  Phase  should  be  used.  This  will  allow 
high  technical  risk  items  to  be  tested  early  and  subsequent  tests  to 
be  incorporated  into  the  hardware  as  the  system  concept  has  been 
demonstrated  as  feasible. 
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25.3.2  Component  and  Subsystem  Test  Plans 

Ensure  a  viable  component  and  subsystem  test  plan.  Studies 
show  that  almost  all  component  failures  will  be  the  kind  that  cannot 
be  easily  detected  or  prevented  in  full  system  testing.  System  failure 
must  be  detected  and  fixed  in  the  componenVsubsystem  stage  as  detecting 
failure  only  at  the  operational  test  level  makes  the  cost  of  correcting 
such  failures  very  high. 

25.3.3  Phasing  of  DT&E  and  IOT&E 

Problems  that  become  apparent  in  the  operational  testing  can 
often  be  evaluated  much  more  quickly  with  the  instrumented  DT&E  hardware. 
The  integrated  test  plan  should  allow  for  time  and  money  to  investigate 
test  failures  and  make  provisions  for  eliminating  the  rause  of  the  failures 
before  other  similar  tests  take  place. 

25.3.4  Scheduling  IOT&E  To  Include  System  Interfaces  With  Other  Systems 

Whenever  possible,  the  IOT&E/FOT&E  of  a  weapon  system  should 
be  planned  to  include  other  systems  which  must  have  a  technical  interface 
with  the  new  system.  For  example,  the  missile  should  be  tested  on  most 
of  the  platforms  for  which  they  are  programmed. 

25.4  RESOURCES  FOR  TESTING 

25.4.1  Identification  Of  Test  Resources  and  Instrumentation 

As  early  as  possible,  but  not  later  than  the  start  of  the 
Full-Scale  Development  phase,  the  test  facilities  and  instrumentation 
requirements  to  conduct  operational  tests  should  be  identified,  along 
with  a  tentative  schedule  of  test  activities.  This  information  is  recorded 
in  the  Test  and  Evaluation  Master  Plan  (TEMP)  and  Service  test  resource 
documentation. 

25.4.2  Requirement  For  Multi  service  OT&E 

Multi  service  OT&E  should  be  considered  for  those  weapon  systems 
which  require  new  operational  concepts  involving  other  Services.  If 
multiservice  testing  is  used,  an  analysis  of  the  impact  of  demonstration 
on  time  and  resources  needed  to  execute  the  multiservice  tests  should 
be  conducted  before  the  Milestone  II  decision. 

25.4.3  Military  Construction  Program  Facilities 

Some  programs  cannot  be  tested  without  Military  Construction 
Program  facilities.  To  construct  these  facilities  will  require  long 
lead  times  therefore,  early  planning  must  be  done  to  ensure  that  the 
facilities  will  be  ready  when  required. 
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25.4.4  Test  Sample  Size 


The  primary  basis  for  tne  test  sample  size  is  usually  based 
on  one  or  more  of  the  following: 

o  Analysis  of  test  objectives; 

o  Statistical  significance  of  test  results  at  some  specified 
confidence  level; 

o  Availability  of  test  vehicles,  items,  etc.; 
o  Support  resources  or  facilities  available;  or 
o  Time  available  for  the  test  program. 

25.4.5  Test  Termination 

One  should  not  hesitate  to  terminate  a  test  prior  to  its 
completion  when  it  becomes  clear  that  the  main  objective  of  the  test 
is  unachievable  (because  of  hardware  failure,  unavailability  of  resources, 
etc.),  or  that  additional  samples  will  not  change  the  outcome  and 
conclusions  of  the  test. 

25.5  COST 

25.5.1  Budgeting  For  Test 

The  DCP,  TEMP,  and  later  budgeting  documents  should  be  regularly 
reviewed  to  ensure  that  there  are  adequate  identified  funds  for  testing, 
relative  to  development  and  fabrication  funds. 

25.5.2  Funds  For  Correction  Of  Faults  Found  In  Testing 

The  DCP,  TEMP  and  later  budgeting  documents  need  careful  scrutiny 
to  ensure  that  there  are  adequate  contingency  funds  to  cover  correction 
of  difficulties  at  a  level  that  matches  the  Industry/Government  experience 
on  the  contract.  (Testing  to  correct  deficiencies  found  during  testing 
without  sufficient  funding  for  proper  correction,  results  in  band-aid 
approaches  which  ultimately  require  corrections  at  a  later  and  more 
expensive  time  period.) 

25.6  PERFORMANCE  AND  OPERATIONAL  ISSUES 

25.6.1  Proof  Of  Performance  Of  Human  Factors  Concepts 

At  an  appropriate  time  in  Concept  Exploration/Definition  or 
Concept  Demonstration/Validation  phases,  T&E  should  authenticate  the 
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human  factors  concepts  embodied  in  the  proposed  systems  design,  examining 
questions  of  safety,  comfort,  man-machine  interfaces,  as  well  as  the 
number  and  skill  of  personnel  required. 

25.6.2  Test  Planning 

A  summary  of  important  test  planning  items  that  were  identified 
by  the  DSB  are  provided  below: 

o  Ensure  that  the  whole  system,  including  the  system  user 

personnel,  are  tested.  Realistically  test  the  complete  system, 
including  hardware,  software,  people  and  all  interfaces.  Get 
user  involved  from  the  start  and  understand  user  limitations. 

o  Ascertain  that  sufficient  time  and  test  articles  are  planned. 

When  the  technology  is  stressed,  the  higher  risks  require  more 
test  articles  and  time. 

o  In  general,  parts,  subsystems  and  systems  should  be  proven 

in  that  order  before  incorporating  them  into  the  next  higher 
assembly  for  more  complete  tests.  The  instrumentation  should 
be  planned  to  permit  diagnosis  of  trouble. 

o  Major  tests  should  never  be  repeated  without  an  analysis  of 

failure  and  corrective  action.  Allow  for  delays  of  this  nature. 

25.7  SPECIFIC  WEAPON  SYSTEMS  TESTING  CHECKLIST 

The  DSB  report  is  the  result  of  the  study  of  past  major  weapon 
systems  acquisitions.  It  was  hoped  that  this  study  would  enhance  the 
testing  community's  understanding  of  the  role  which  test  and  evaluation 
has  had  in  identifying  system  problems  during  the  acquisition  process. 
In  the  foreword  of  the  DSB  study,  the  authors  made  this  statement  about 
including  the  obvious  testing  activity  in  their  checklist: 

The  T&E  expert  in  reading  this  volume  will  find  many 
precepts  which  will  strike  him  as  of  this  type.  These 
items  are  included  because  examples  were  found  where 
even  the  obvious  has  been  neglected,  not  because  of 
incompetence  or  lack  of  personal  dedication  by  the 
people  in  charge  of  the  program,  but  because  of  financial 
and  temporal  pressures  which  forced  competent  managers 
to  compromise  on  their  principles.  It  is  hoped  that 
the  inclusion  of  the  obvious  will  prevent  repetition 
of  the  serious  errors  which  have  been  made  in  the  past 
when  such  political,  economical  and  temporal  pressures 
have  forced  project  managers  to  depart  from  the  rules 
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of  sound  engineering  practices...  In  the  long  run, 
taking  short  cuts  during  T&E  to  save  time  and  money 
will  result  in  significant  increases  in  the  overall 
costs  of  the  programs  and  in  a  delay  of  delivery  of 
the  corresponding  weapon  systems  to  combatant  forces. 


25.7.1  Aircraft  Systems 

25.7.1.1  Concept  Exploration/Definition  Phase 

o  Test  Program/Total  Costs.  Prior  to  Milestone  I,  all  the  phases 
of  the  aircraft  test  program  should  be  considered  so  that  the  total 
costs  and  the  development  schedules  include  consideration  of  all 
likely  activities  in  the  overall  program. 

o  Test  Facilities  and  Instrumentation.  Prior  to  Milestone  I, 
the  test  facilities  and  instrumentation  requirements  to  conduct  tests 
should  be  generally  identified  along  with  a  tentative  schedule  of 
test  activities. 

o  Test  Resources  and  Failures.  Ensure  that  there  are  adequate 
funds,  reasonable  time  and  an  acceptable  number  of  aircraft  planned 
for  the  various  test  program  phases  and  that  these  make  provisions 
for  the  occurrence  of  failures. 

o  System  Interfaces.  Consider  all  aircraft  system  interfaces, 
their  test  requirements  and  probable  costs  at  the  outset  of  the  Concept 
Exploration/Definition  phase. 

o  Major  Weapon  Subsystems.  If  the  aircraft  system  relies  on  the 
successful  development  of  a  specific  and  separately  funded  major 
weapon  (such  as  a  gun  or  missile)  in  order  to  accomplish  its  primary 
mission,  this  major  subsystem  should  be  developed  and  test  concurrently 
with  or  prior  to  the  aircraft. 

o  Propulsion  System.  If  the  aircraft  program  is  paced  by  the 
propulsion  system  development,  an  early  advanced  development  project 
for  the  propulsion  may  be  appropriate  for  a  new  concept. 

o  Operational  Scenario.  A  conceptual  operational  scenario  for 
operation  and  use  of  the  aircraft  should  be  developed  so  that  general 
test  plans  can  be  designed.  This  should  include  purpose,  roles  and 
missions,  threats,  operating  environments,  logistics  and  maintenance, 
and  basing  characteristics.  The  potential  range  of  values  on  these 
aspects  should  be  stated. 

o  Evaluation  Criteria.  Develop  evaluation  criteria  to  be  used 
for  the  selection  of  the  final  aircraft  system  design. 
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o  Untried  Elements.  The  aircraft  development  program  should  include 
conclusive  testing  to  eliminate  uncertainties  of  the  untried  elements. 

o  Brassboard  Avionics  Tests.  The  use  of  brassboard  or  modified 
existing  hardware  to  "prove"  that  the  concept  will  work  should  be 
seriously  scrutinized  to  ensure  that  the  demonstrations  and  tests 
are  applicable. 

o  Nuclear  Weapons  Effects.  The  subject  of  nuclear  weapons  effects 
should  be  addressed  in  the  test  concept  for  all  aircraft  weapons 
systems  where  operational  suitability  dictates  that  survivable  exposure 
to  nuclear  weapons  effects  is  a  requirement. 

25.7.1.2  Concept  Demon stration/Validat ion  Phase 

o  By  the  end  of  the  validation  phase,  test  and  evaluation  plans 
and  test  criteria  should  be  established  so  there  is  no  question  as 
to  what  constitutes  a  successful  test  and  what  performance  is  required. 

o  Milestones  and  Goals.  Assure  an  integrated  system  test  plan 
that  pre-establishes  milestones  and  goals  for  easy  measurement  of 
program  progress  at  a  later  time. 

o  Operating  Concept  and  Environment.  The  operational  concept 
and  it  environments  in  which  the  aircraft  will  be  expected  to  operate 
and  to  be  tested  in  OT&E  should  be  specified. 

o  Test  Program  Building  Blocks.  In  the  validation  phase, 
demonstrate  that  the  high  risk  technology  is  in  hand  and  in  planning 
the  full-scale  development  test  program  ensure  components  and  the 
subsystems  are  adequately  qualified  for  incorporation  into  the  system 
tests. 

o  Technology  Concepts.  Each  concept  to  be  used  in  the  aircraft 
system  (e.g.,  aerodynamics,  structures,  propulsion)  should  be 
identified  and  coded  according  to  prior  application,  prior  to  future 
research;  tests  for  each  of  the  concepts  should  be  specified  with 
the  effect  of  failure  identified. 

o  DT&E  /  OT&E  Plan.  The  aircraft  DT&E/OT&E  test  plan  should  be 
reviewed  to  ensure  it  includes  ground  and  flight  tests  necessary 
to  safely  and  effectively  develop  the  system. 

o  Test  Failures.  Make  T&E  plans  assuming  failures--they  are 
inevitable. 

o  Multi  service  Testing.  When  a  new  aircraft  development  program 
requires  joint  testing  during  OT&E,  prior  to  Milestone  II,  the  test 
plan  should  include  the  type  of  tests  and  resources  required  from 
other  activities  and  Services. 
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o  Traceability.  The  aircraft  development  and  test  program  should 
be  designed  and  scheduled  in  such  a  way  that  if  trouble  arises,  the 
source  of  the  trouble  can  be  traced  back  through  the  lab  tests  and 
the  analytical  studies. 

o  Competitive  Prototype  Tests.  When  a  competitive  prototype  test 
program  is  used,  the  aircraft  should  be  compared  on  the  basis  of 
the  performance  of  critical  mission  using  both  test  and  operational 
crews. 

o  Prototype  Similarity  To  Development  and  Production  Aircraft. 
A  firm  determination  should  be  made  of  the  degree  of  similarity  of 
the  winning  prototype  (in  a  competitive  prototype  program)  to  the 
development  and  production  aircraft  in  order  that  test  results  derived 
from  the  prototype  in  the  interim  period  prior  to  availability  of 
the  engineering  development  aircraft  can  be  utilized  most  effectively. 

o  Prototype  Tests.  The  prototype  aircraft  test  data  should  be 
used  to  determine  where  emphasis  should  be  placed  in  the  engineering 
development  program. 

o  Inlet  /  Engine  /  Nozzle  Match.  The  aircraft  test  program  should 
provide  for  early  and  adequate  inlet/engine/nozzle  match  through 
a  well  planned  test  program  with  time  programming  for  corrections. 

o  Subsystem  Tests.  There  should  be  a  balanced  program  for  the 
aircraft  subsystem  tests. 

o  Propulsion  System.  If  the  aircraft  is  paced  by  the  propulsion 
systems  development,  an  early  advanced  development  project  for  the 
propulsion  may  be  appropriate  for  a  new  concept. 

o  EMI  Testing.  Full  scale  aircraft  systems  tests  in  an  an echoic 
chamber  are  desirable  for  s^me  aircraft. 

o  Parts  Interchange.  Early  plans  should  provide  for  tests  where 
theoretically  identical  parts,  particularly  in  avionics,  are 
interchanged  to  ensure  that  the  aircraft  systems  can  be  maintained 
in  readiness. 

o  Human  Factors  Demonstration.  Ensure  adequate  demonstration 
of  human  factors  is  considered  in  the  test  plan. 

o  Military  Preliminary  Evaluation.  Adequate  resources  should 
be  scheduled  for  the  aircraft  Military  Preliminary  Evaluation  (MPE) 
and  a  positive  program  should  exist  for  the  utilization  of  MPE 
information  at  the  time  of  OT&E. 
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o  User  Participation.  It  is  imperative  that  the  operational  command 
actively  participate  in  the  DT&E  phase  to  ensure  that  the  user  needs 
are  represented  in  the  development  of  the  system. 

o  Maintenance  and  Training  Publications.  The  aircraft  development 
program  should  provide  for  concurrent  training  of  crews  and  for  the 
preparation  of  draft  technical  manuals  to  be  used  by  IOT&E  maintenance 
and  operating  crews. 

o  R&D  Completion  Prior  To  IOT&E.  The  testing  plans  should  ensure 
that  before  an  aircraft  system  is  subjected  to  IOT&E,  the  subsystems 
essential  to  the  basic  mission  have  completed  R&D. 

25.7.1.3  Full-Scale  Development  Phase 

o  Test  Design.  Test  programs  should  be  designed  to  have  a  high 
probability  of  identifying  major  deficiencies  early,  during  the  DT&E 
and  IOT&E. 

o  Data  for  Alternate  Scenarios.  Maximize  the  utility  of  the  test 
data  gathered  by  careful  attention  to  testing  techniques;  aircraft 
instrumentation;  range  instrumentation;  and  data  collection,  reduction 
and  storage. 

o  Test  Milestones.  Development  programs  should  be  built  around 
testing  milestones,  not  calendar  dates. 

o  Production  Engineering  Influence  on  R&D  Hardware.  Encourage 
that  production  philosophy  and  production  techniques  be  brought  into 
an  early  phase  of  the  design  process  for  R&D  hardware  to  the  maximum 
extent  practical. 

o  Running  Evaluation  of  Tests.  Ensure  that  running  evaluations 
of  test  are  conducted.  If  it  becomes  clear  that  test  objectives 
are  unattainable  or  that  additional  samples  will  not  change  the  test 
outcome,  ensure  procedures  are  established  for  terminating  the  test. 

o  Simulation.  Analysis  and  simulation  should  be  conducted,  where 
practicable,  before  each  phase  of  development  flight  testing. 

o  Avionics  Mock-up.  Encourage  use  of  a  complete  avionics  system 
installed  in  a  mock-up  of  the  appropriate  section  or  sections  of 
the  aircraft. 

o  Escape  Systems  Testing.  Ensure  the  aircrew  escape  system  is 
thoroughly  tested  with  particular  attention  to  redundant  features, 
such  as  pyrotechnic  firing  channels. 
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o  Structural  Testing.  Assure  that  fatigue  testing  is  conducted 
on  early  production  airframes.  Airframe  production  should  be  held 
to  a  low- rate  until  satisfactory  progress  is  shown  in  these  tests. 

o  Gun  Firing  Tests.  All  forms  of  ordnance,  and  especially  those 
which  create  gases  must  be  fired  from  the  aircraft  for  external  effects 
(blast  and  debris),  internal  effects  (shock),  and  effects  on  the 
propulsion  (inlet  composition  or  distribution). 

o  Post  Stall  Characteristics.  Special  attention  is  warranted 
on  the  post  stall  test  plans  for  DT&E  and  OT&E. 

o  Subsystem  Performance  History.  During  DT&E  and  IOT&E  of  aircraft, 
ensure  a  performance  history  of  each  subsystem  of  the  aircraft  will 
be  kept. 

o  Flight  Deficiency  Reporting.  Composition  of  flight  deficiencies 
reporting  by  aircrews,  particularly  those  pertaining  to  avionics, 
should  be  given  special  attention. 

o  Crew  Limitations.  Ensure  aircrew  limitations  are  included  in 
the  tests. 

o  Use  of  Operational  Personnel.  Recommend  experienced  operational 
personnel  help  in  establishing  measures  of  effectiveness  and  in  other 
operational  test  planning.  In  conducting  OT&E,  use  typical  operational 
aircrews  and  support  personnel. 

o  Role  of  the  User.  Ensure  that  users  participate  in  the  T&E 
phase  so  that  their  needs  are  represented  in  the  development  of  the 
system  concept  and  hardware. 

o  Crew  Fatigue  and  System  Effectiveness.  In  attack  aircraft 
operational  testing,  and  particularly  in  attack  helicopter  tests 
where  vibration  is  a  fatiguing  factor,  ascertain  that  the  tests  include 
a  measure  of  degradation  over  time. 

o  Time  Constraints  on  Crews.  Detailed  operational  test  plans 
should  be  evaluated  to  determine  that  the  test-imposed  conditions 
on  the  crew  do  not  invalidate  the  applicability  of  the  data  so 
collected. 

o  Complete  Basic  DT&E  Before  Starting  OT&E.  Before  the  weapon 
system  is  subjected  to  IOT&E,  all  critical  subsystems  should  have 
completed  basic  DT&E  with  significant  problems  solved. 

o  Realism  in  Testing.  Ascertain  that  final  DT&E  system  tests 
and  IOT&E  flight  tests  are  representative  of  operational  conditions. 
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o  Test  All  Profiles  and  Modes.  Tests  should  be  conducted  to 
evaluate  all  planned  operational  flight  profiles  and  all  primary 
and  back-up,  degraded  operating  modes. 

o  Update  of  Operational  Test  Plans.  Ensure  operational  test  plans 
are  reviewed  and  updated  as  needed  to  make  them  relevant  to  evolving 
concepts. 

o  Conduct  IOT&E  Early.  Ensure  operational  suitability  tests  are 
planned  to  attempt  to  identify  operational  deficiencies  of  new  systems 
quickly  so  that  fixes  can  be  developed  and  tested  before  large  scale 
production. 

o  Missile  Launch  Tests.  Review  the  final  position  fix  planned 
before  launching  inertial  guided  air-to-surface  missiles. 

o  Mission  Completion  Success  Probability.  Mission  completion 
success  probability  factors  should  be  used  to  measure  progress  in 
the  aircraft  test  program. 

25.7.1.4  Full-Rate  Production  Phase 

o  Operational  Test  Realism.  Ascertain  operational  testing  is 
conducted  under  realistic  conditions. 

o  Design  OT&E  For  Less  Than  Optimal  Condition.  Structure  the 
OT&E  logistical  support  for  simulated  combat  conditions. 

o  New  Threat.  Be  alert  to  the  need  to  extend  the  OT&E  if  a  new 
threat  shows  up. 

o  Certification  of  Ordnance.  Assure  that  ordnance  to  be  delivered 
by  an  aircraft  is  certified  for  the  aircraft. 

o  Inadvertent  Influence  of  Test.  OT&E  plans  should  provide  measures 
of  ensuring  that  actions  by  observers  and  umpires  do  not  unwittingly 
influence  trial  outcome. 

o  Deficiencies  Discovered  In-Service.  Be  aware  that  in-service 
operations  of  an  aircraft  system  will  surface  deficiencies  which 
extensive  follow-on  OT&E  probably  would  not  uncover. 

o  Lead  The  Fleet.  Accelerated  service  test  of  a  small  quantity 
of  early  production  aircraft  is  advisable  during  follow-on  OT&E 
thereafter. 
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25.7.2  Missile  Systens 

25.7.2.1  Concept  Exploration/Definition  Phase 

o  Weapon  System  Interfaces.  Consider  significant  weapon  system 
interfaces,  their  test  requirements  and  probable  costs  at  the  outset 
of  the  Concept  Exploration/Definition  Phase.  Ensure  that  the  program 
plan  assembled  before  Milestone  I  includes  an  understanding  of  the 
basic  test  criteria  and  broad  test  plans  for  the  whole  program. 


o  Number  of  Test  Missiles.  Ensure  that  there  is  sufficient  time 
and  a  sufficient  number  of  test  articles  to  support  the  program  through 
its  various  phases.  Compare  the  program  requirements  with  past  missile 
programs  of  generic  similarity.  If  there  is  substantial  difference, 
then  adequate  justification  should  be  provided.  The  DT&E  period 
on  many  programs  has  had  to  be  extended  as  much  as  50  percent. 

o  T&E  Gap.  A  test  and  evaluation  gap  has  been  experienced  in 
some  missile  programs  between  the  time  when  testing  with  R&D  hardware 
was  completed  and  the  time  when  follow-on  operational  suitability 
testing  was  initiated  with  production  hardware. 

o  Feasibility  Tests.  Ensure  experimental  test  evidence  is  available 
to  indicate  the  feasibility  of  the  concept  and  the  availability  of 
the  technology  for  the  system  development. 

o  Evaluation  of  Conceptual  and  Validation  Tests.  Results  of  tests 
conducted  during  the  Concept  Exploration/Definition  and  the  Concept 
and  Demcnstration/Validation  phases,  which  most  likely  have  been 
conducted  as  avionics  brassboard,  breadboard,  or  as  modified  existing 
hardware,  should  be  evaluated  with  special  attention. 

o  Multi  service  Testing  Plans.  When  a  new  missile  development 
program  requires  multi  service  testing  during  OT&E,  the  test  plan 
should  include  the  type  of  tests  and  resources  required  from  other 
activities  and  services. 

o  Test  Facilities  and  Instrumentation  Requirements.  Before 
Milestone  I  the  test  facilities  and  instrumentation  requirements 
to  conduct  tests  should  be  generally  identified  along  with  a  tentative 
schedule  of  test  activities. 


25.7.2.2  Concept  Oemonstration/Validation  Phase 

o  Establish  Test  Criteria.  By  the  end  of  the  Concept 
Demonstration/Validation  phase,  test  criteria  should  be  established 
so  that  there  is  no  question  as  to  what  will  constitute  a  successful 
test  and  what  performance  is  expected. 
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o  Human  Factors.  Ensure  that  the  test  plan  includes  adequate 
demonstration  of  human  factors  consideration. 

o  Instrumentation  Diagnostic  Capability  and  Compatibility. 
Instrumentation  design  with  adequate  diagnostic  capability  and 
compatibility  in  both  DT&E  and  IOT&E  phases  is  essential. 

o  Provisions  for  Test  Failures.  DT&E  and  OT&E  plans  should  make 
provisions  for  the  occurrence  of  failures. 

o  Integrated  Test  Plan.  Assure  an  integrated  system  test  plan 
that  pre-establishes  milestones  and  goals  for  easy  measurement  of 
program  progress  at  a  later  time. 

o  Test  and  Evaluation  Requirements.  Ensure  that  the  test  and 
evaluation  program  requirements  are  firm  before  approving  an  R&E 
test  program.  Many  missile  programs  have  suffered  severe  cost  impacts 
as  a  result  of  this  deficiency.  The  test  plan  must  include  provisions 
to  adequately  test  those  portions  of  the  operational  envelope  which 
stress  the  system  including  backup  and  degraded  operational  modes. 

o  Personnel  Training  Plans.  Ensure  that  adequate  training  and 
certification  plans  for  test  personnel  have  been  developed. 

o  T&E  Reporting  Format.  Include  a  T&E  reporting  format  in  the 
program  plan.  Attention  must  be  given  to  the  reporting  format  in 
order  to  provide  a  consistent  basis  for  test  and  evaluation  throughout 
the  program  life  cycle. 

o  Program-to-Program  Crosstalk.  Encourage  program- to-program 
T&E  crosstalk.  Test  and  evaluation  problems  and  their  solutions 
as  one  program  provide  a  valuable  index  of  lessons  learned  and 
techniques  for  problem  resolution  on  other  programs. 

o  Status  of  T&E  Offices.  Ensure  that  Test  and  Evaluation  offices 
have  the  same  stature  as  other  major  elements,  reporting  to  the  program 
manager  or  director.  It  is  important  that  the  test  and  evaluation 
component  of  the  system  program  office  have  organizational  status 
and  authority  equal  to  configuration  management,  program  control, 
system  engineering,  etc. 

o  Measurement  of  Actual  Environments.  Thorough  measurements  should 
be  made  to  define  and  understand  the  actual  environment  in  which 
the  system  components  must  live  during  the  captive,  launch  and 
in-flight  phases. 


25-13 


o  Thoroughness  of  Laboratory  Testing.  Significant  time  and  money 
will  be  saved  if  each  component,  each  subsystem,  and  the  full  system 
are  all  tested  as  thoroughly  as  possible  in  the  laboratory. 

o  Contract  Form.  The  contract  form  can  be  extremely  important 
to  the  T&E  aspects.  In  one  program,  the  contract  gave  the  contractor 
full  authority  to  determine  the  number  of  test  missiles,  and  in  another 
the  contract  incentive  resulted  in  the  contractor  concentrating  tests 
on  one  optimum  profile  to  satisfy  the  incentive  instead  of  developing 
the  performance  throughout  important  areas  of  the  envelope. 

o  Participation  of  Operational  Command.  It  is  imperative  that 
the  operational  command  actively  participate  in  the  DT&E  phase  to 
ensure  that  the  user  needs  are  represented  in  the  development  of 
the  system. 


25.7.2.3  Full-Scale  Development  Phase 

o  Production  Philosophy  and  Techniques.  Encourage  that  production 
philosophy  and  production  techniques  be  brought  into  an  early  phase 
of  the  design  process  for  R&D  hardware  to  the  maximum  extent  practical. 
There  are  many  missile  programs  in  which  the  components  were  not 
qualified  until  the  missile  was  well  into  production 

o  Operational  Flight  Profiles.  Tests  should  be  conducted  to 
evaluate  all  planned  operational  flight  profiles  and  all  primary 
and  back-up  degraded  operating  modes. 

o  Failure  Isolation  and  Responsive  Action.  Does  the  system  test 
plan  provide  for  adequate  instrumentation  so  that  missile  failures 
can  be  isolated  and  fixed  before  the  next  flight. 

o  Responsive  Actions  for  Test  Failures.  Encourage  a  closed  loop 
reporting  and  resolution  process  which  assures  that  each  test  failure 
at  every  level  is  closed  out  by  appropriate  action,  i.e.,  redesign, 
procurement,  retest,  etc. 

o  Plan  Tests  of  Whole  System.  Plan  tests  of  the  whole  system 
including  proper  phasing  of  the  platform  and  supporting  gear,  the 
launcher,  the  missile,  and  the  user's  participation. 

o  Determination  of  Component  Configuration.  Conditions  and 
component  configuration  during  development  tests  should  be  determined 
by  the  primary  objectives  of  that  test.  Whenever  a  non-operational 
configuration  is  dictated  by  early  test  requirements,  tests  should 
not  be  challenged  by  the  fact  that  configuration  is  not  operational. 
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o  Testing  of  Software.  Test  and  evaluation  should  ensure  that 

software  products  are  tested  appropriately  during  each  phase.  Software 
has  often  been  developed  more  as  an  add-on  than  as  an  integral  part 
of  the  overall  system.  Software  requirements  need  the  same 

consideration  as  hardware  requirements  in  the  Concept 
Demonstrati on/Val idation  Phases. 

o  Range  Safety  Dry  Runs.  Ensure  the  test  plan  includes  adequate 

test  program/range  safety  dry  runs.  The  government  test  ranges  have 

to  provide  facilities  to  safely  test  many  different  projects. 

o  Assemblies/Subsystems  Special  Requirements. 

Seekers  and  tracking  devices, 

Propulsion  subsystems. 

Connectors  and  their  related  hardware. 

Lanyard  assemblies,  and 

Safing,  arming,  fuzing  and  other  ordnance  devices. 

o  Review  of  Air-To-Surface  Missile  Test  Position  Fixes.  Review 
the  final  position  fix  planned  before  launching  ASMs.  There  are 

instances  in  which  the  operational  test  of  air  launched  missiles 
utilized  artificial  position  fixes  just  prior  to  missile  launch. 

o  Operator  Limitations.  Ensure  operator  limitations  are  included 

in  the  tests.  Most  tactical  missiles,  especially  those  used  in  close 
support,  require  visual  acquisition  of  the  target  by  the  missile 
operator  and/or  an  air/ground  controller. 

o  Test  Simulations  and  Dry  Runs.  Plan  and  use  test  simulations 

and  dry  runs.  Dry  runs  should  be  conducted  for  each  new  phase  of 
testing.  Simulation  and  other  laboratory  or  ground  testing  should 
be  conducted  to  predict  the  specific  test  outcome.  The  "wet  run" 
test  should  finally  be  run  to  verify  the  test  objectives.  Evaluation 
of  the  simulation  versus  the  actual  test  results  will  help  to  refine 
the  understanding  of  the  system. 

o  Component  Performance  Records.  Keep  performance  records  on 

components.  There  are  many  examples  in  missiles  programs  which  have 
required  stock  sweeps  that  are  associated  with  flight  failures  and 
aging  testing  programs. 

o  Tracking  of  Test  Data.  Ensure  the  test  program  tracks  data 

in  a  readily  usable  manner.  Reliability  and  performance  evaluations 
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of  a  missile  system  should  break  down  the  missile's  activity  into 
at  least  the  following  phases: 

Pre-launch  including  captive  carry  reliability. 

Launch, 

In-flight,  and 
Accuracy/fuzing. 

o  Updating  of  IOT&E  Planning.  Periodically  update  military 
preliminary  evaluation  (MPE)  and  IOT&E  planning  during  the  early 
R&D  phase.  Few  missile  system  programs  have  had  adequate  user 
participation  with  the  desirable  continuity  of  personnel  to  minimize 
the  problems  of  transition  from  DT&E  to  OT&E  to  deployment/utilization. 

o  Instrumentation  Provisions  in  Production  Missiles.  Encourage 
built-in  instrumentation  provisions  in  production  missiles. 

o  Constraints  on  Missile  Operator.  Detailed  test  plans  should 
be  evaluated  to  determine  that  the  test  imposed  constraints  on  the 
missile  operator  do  not  invalidate  the  applicability  of  the  data 
so  collected. 

o  Problem  Fixes  Before  Production.  Ensure  operational  suitability 
tests  identify  operational  deficiencies  of  new  systems  quickly  so 
that  fixes  can  be  developed  and  tested  before  large  scale  production. 

o  Flight  Tests  Representative  of  Operations.  Ascertain  that  final 
DT&E  system  tests  and  IOT&E  flight  tests  are  representati ve  of 
operational  flights.  Some  ballistic  missile  R&E  programs  have  shown 
very  high  success  rates  in  R&E  flight  test;  however,  when  the  early 
production  systems  were  deployed,  they  exhibited  a  number  of 
unsatisfactory  characteristics  such  as  poor  alert  reliability  and 
poor  operational  test  flight  reliability. 

25.7.2.4  Full  Rate  Product ion/Deployment  Phase 

o  System  Interfaces  in  Operational  Test.  Ensure  the  primary 
objective  of  an  operational  test  is  to  obtain  measurements  on  the 
overall  performance  of  the  weapon  system  when  it  is  interfaced  with 
those  systems  required  to  operationally  use  the  weapons  system. 

o  Realistic  Conditions  for  Operational  Testing.  Ascertain 
operational  testing  is  conducted  under  realistic  combat  conditions. 
This  means  that  the  offense/defense  battle  needs  to  be  simulated 
in  some  fashion  before  the  evaluation  of  the  weapon  system  can  be 
considered  completed.  Whether  this  exercise  is  conducted  within 
a  single  service  (as  in  the  test  of  a  surface-to-surface  anti-tank 
missile  against  tanks)  or  between  services  (as  in  the  test  of  an 
air-to-surface  missile  against  tanks  with  anti-aircraft  protection), 
the  plans  for  such  testing  should  be  formulated  as  part  of  the  system 
development  plan. 
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o  Testing  of  all  Operational  Modes.  Assure  the  follow-on  OT&E 
plan  includes  tests  of  any  operational  modes  not  previously  tested 
in  IOT&E .  All  launch  modes,  included  degraded,  backup  modes,  should 
be  tested  in  the  follow-on  OT&E  because  the  software  interface  with 
the  production  hardware  system  should  be  thoroughly  evaluated, 
otherwise  small,  easy  to  fix  problems  might  preclude  launch. 

o  Extension  of  the  OT&E  for  New  Threats.  Be  alert  to  the  need 
to  extend  the  OT&E  if  a  new  threat  shows  up.  Very  few  missile  programs 
perform  any  kind  of  tests  relatable  to  evaluating  system  performance 
against  current  threats,  let  alone  new  threats. 

o  "Lead-the-Fleet"  Production  Scheduling.  "Lead-the-Fleet"  missile 
scheduling  and  tests  should  be  considered. 

o  Test  Fixes.  Test  fixes  resulting  from  earlier  operational 
testing.  Following  initial  operational  tests  which  identify  problem 
areas  in  missiles,  Follow-on  OT&E  should  be  alert  in  these .  areas 
with  the  primary  intent  of  investigating  the  adequacy  of  the  fixes 
incorporated,  particularly  if  the  IOT&E  did  not  run  long  enough  +  ~ 
test  the  fixes. 

o  OT&E  Feedback  to  Acceptance  Testing.  Ensure  OT&E  results  are 
quickly  fed  back  to  influence  early  production  acceptance  testing. 
Production  acceptance  testing  is  probably  the  final  means  /the 
government  will  normally  have  to  ensure  the  product  meets 
specifications.  That  early  acceptance  testing  could  be  influenced 
favorably  by  a  quick  feedback  from  Follow-on  OT&E  to  acceptance  testing 
is  exemplified  by  a  current  ASM  program  where  production  has  reached 
peak  rates  and  the  OT&E  has  hot  been  completed. 


25.7.3  Command  and  Control  Systems 

25.7.3.1  Concept  Exploration/Definition  Phase 

o  Conceptual  Test  Philosophy.  T&E  planners  must  understand  the 
nature  of  Command  and  Control  systems  early  in  the  Concept 
Exploration/Definition  Phase.  In  a  complex  Command  and  Control  system, 
a  total  systems  concept  has  to  be  developed  at  the  beginning.  Total 
systems  life  cycle  must  be  analyzed  so  the  necessary  requirement 
for  the  design  can  be  established. 

o  The  Importance  of  Software  Testing.  Testers  should  recognize 
that  software  is  a  pacing  item  in  Command  and  Control  systems 
development. 
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o  Software  Test  Scheduling  -  Contractors'  Facilities.  Provision 
should  be  made  for  inclusion  of  software  T&E  during  each  phase  of 
C&C  systems'  acquisition.  Availability  of  contractors'  facilities 
should  be  considered. 

o  Evaluation  of  Exploratory  Development  Tests.  Care  should  be 
exercised  in  evaluating  results  of  tests  conducted  during  exploratory 
development  of  Command  and  Control  Systems.  Results  of  tests  conducted 
during  exploratory  development  and  which  most  likely  have  been 
conducted  on  brassboard,  breadboard,  or  modified  existing  hardware 
should  be  evaluated  with  special  attention. 

o  Feasibility  Testing  for  Field  Compilers.  Early  test  planning 
should  allow  for  simulation  of  the  computer  system  to  test  for  field 
use  of  compilers,  where  applicable. 

o  Evaluation  of  Test  Plan  Scheduling.  Milestones  should  be 
event-oriented,  not  calendar-oriented. 

o  Type  Personnel  Needs  -  Effects  on  T&E.  A  mix  of  personnel  with 
different  backgrounds  affecting  T&E  is  required. 

o  Planning  for  Joint  Service  OT&E  Before  Milestone  I.  Joint  Service 
Operation  Test  and  Evaluation  should  be  considered  for  Command  and 
Control  systems. 

25.7.3.2  Concept  Demonstration/Validation  Phase 

o  Test  Prototypes.  In  Command  and  Control  Systems,  prototypes 
must  reasonably  resemble  final  hardware  configuration  from  a  functional 
use  standpoint.  When  high  technical  risk  is  present,  development 
should  be  structured  around  the  use  of  one  or  more  test  prototypes 
designed  to  prove  the  system  concept  under  realistic  operational 
conditions  before  proceeding  to  engineering  development. 

o  Test  Objectives  -  Critical  Issues.  In  addition  to  addressing 
critical  technical  issues,  T&E  objectives  during  the  Concept 
Demonstration/Validation  Phase  should  address  the  functional  issues 
of  a  Command  and  Control  system. 

o  Real  Time  Software  -  Demonstration  of  "Application  Patches". 
Tests  of  real  time  Command  and  Control  systems  should  include 
demonstrations  of  interfaces  whereby  locally  generated  application 
patches  are  brought  into  being. 

o  Independent  Software  Test-User  Group.  An  independent  test-user 
software  group  is  needed  during  early  software  qualification  testing. 
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o  System  Interfaces.  Critical  attention  should  be  devoted  to 
testing  interfaces  with  other  C&C  systems  and  to  interfaces  between 
subsystems.  Particular  attention  should  be  devoted  to  interfaces 
with  other  C&C  systems  and  to  the  interfaces  between  sensors  (e.g., 
radars),  communications  systems  (e.g.,  modems),  and  the  specific 
processors  (e.g.,  CPU).  Interface  with  information  processing  C&C 
systems  must  also  address  data  element  and  code  standardization 
problems  if  data  is  to  be  processed  on-line. 

o  Human  Factors.  In  a  C&C  system  human  factors  must  be  considered 
from  the  earliest  prototype  designs  and  testing  provided.  Testing 
should  be  conducted  to  determine  the  most  efficient  arrangement  of 
equipment  from  the  human  factor  standpoint,  e.g.,  displays  should 
be  arranged  so  as  to  be  viewed  from  an  optimum  angle  whenever  possible; 
adequate  maneuvering  room  within  the  installation  constraints  should 
be  allowed  considering  the  number  of  personnel  normally  manning  the 
facility;  and  console-mounted  controls  should  be  so  designed  and 
located  as  to  facilitate  operation,  minimize  fatigue  and  avoid 
confusion. 

o  Degraded  Operations  Testing.  When  the  expected  operational 
environment  of  a  C&C  system  suggests  that  the  system  may  be  operated 
under  less  than  finely  tuned  conditions,  tests  should  be  designed 
to  allow  for  performance  measurements  under  degraded  conditions. 

o  Test  Bed.  The  use  of  a  test  bed  for  study  and  experimentation 
with  new  C&C  systems  is  needed  early  in  the  Concept 
Demonstration/Validation  Phase. 

o  Software- Hardware  Interfaces.  The  software- hardware  interfaces 
with  all  operational  back-up  modes  to  a  new  C&C  system  should  be 
tested  early  in  the  program. 

o  Reproducible  Tests.  Test  plans  should  contain  a  method  for 
allowing  full-load  message  inputs  while  maintaining  reproducible 
test  conditions. 

o  Cost-Effectiveness.  Field  test  data  is  needed  during  the  Concept 
Demonstration/Validation  Phase  for  input  to  cost  effectiveness  analyses 
of  C&C  systems. 

25.7.3.3  Full-Scale  Development  Phase 

o  Acquisition  Strategy.  The  acquisition  strategy  for  the 
system  should: 

Allow  for  sufficient  time  between  the  planned  end 
of  demonstration  testing  and  major  procurement  (as  opposed  to 
limited  procurement)  decisions  so  that  there  is  a  flexibility 
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for  modification  of  plans  which  may  be  required  during  the  test 
phases  of  the  program.  For  instance,  because  insufficient  time 
was  allowed  for  testing  one  recent  C&C  system,  the  program  and 
the  contract  had  to  be  modified  and  renegotiated. 

Be  evaluated  relative  to  constraints  imposed. 

Ensure  that  sufficient  dollars  are  available  not  only 
to  conduct  the  planned  T&E  but  to  allow  for  the  additional  T&E 
which  is  always  required  due  to  failures,  design  changes,  etc. 

o  Problem  Indications.  It  is  important  to  establish  an  early 
detection  scheme  for  management  to  determine  when  a  program 
is  becoming  ill. 

o  Impact  of  Software  Failures.  Prior  to  any  production 
release,  the  impact  of  software  failures  on  overall  system 
performance  parameters  must  be  considered. 

o  Critical  Issues.  IOT&E  should  provide  the  answers  to  some 
critical  issues  peculiar  to  C&C  systems.  Some  of  the  critical 
issues  that  OT&E  of  Command  and  Control  systems  should  answer 
are: 


Is  system  mission  reaction  time  a  significant 

improvement  over  present  systems? 

Is  a  back-up  mode  provided  for  use  when  either  airborne 
or  ground  system  exhibits  a  failure? 

Can  the  system  be  transported  as  operationally  required 
by  organic  transport?  (Consider  ground,  air  and 
amphibious  requirements). 

Is  there  a  special  requirement  for  site  preparation? 
(For  example,  survey,  antenna  siting). 

Can  the  system  be  erected  and  dismantled  in  times 
specified?  Are  these  times  realistic? 

Does  relocation  affect  system  alignment? 

Does  system  provide  for  operation  during  maintenance? 

Can  maintenance  be  performed  on  site  on  non-shelterized 
exposed  subsystems  during  adverse  weather  conditions, 
e.g.,  radars? 


25-20 


o  Displays.  The  display  subsystems  of  a  C&C  system  should  provide 
an  essential  function  to  the  user.  Displays  are  key  subsystems  of 
a  Command  and  Control  system.  They  provide  the  link  that  couples 
the  operator  to  the  rest  of  the  system  and  are  therefore  often  critical 
to  its  success. 

o  Pilot  Test.  A  pilot  test  should  be  conducted  prior  to  IOT&E 
so  that  sufficient  time  is  available  to  make  the  necessary  changes 
to  the  IOT&E  as  dictated  by  the  results  of  the  pilot  test. 

o  Publications  and  Manuals.  It  is  imperative  that  all  system 
publications  and  manuals  be  completed,  reviewed  and  selectively  tested 
under  operational  conditions  prior  to  the  beginning  of  overall  system 
suitability  testing. 

o  Power  Sources.  Mobile  prime  power  sources  are  usually  provided 
as  GFE  and  can  be  a  problem  area  in  testing  C&C  systems. 

o  IOT&E  Reliability  Data.  IOT&E  can  provide  valuable  data  on 
the  operational  reliability  of  a  C&C  system  which  cannot  be  obtained 
through  DT&E. 

o  Subsystem  Tests.  Every  major  subsystem  of  a  C&C  system  should 
have  a  successful  DT&E  prior  to  beginning  of  overall  system  operational 
testing. 

o  Communications.  C&C  systems  must  be  tested  in  the  appropriate 
electromagnetic  environment  to  determine  performance  of  its 
communications  system. 

o  Maintenance.  In  IOT&E,  maintenance  should  include:  A  measurement 
of  the  adequacy  of  the  maintenance  levels  and  the  maintenance 
practices;  an  assessment  of  the  impact  that  the  maintenance  plan 
has  on  the  operational  reliability;  the  accessibility  of  the  major 
components  of  the  system  for  field  maintenance,  e.g.,  are  cables 
and  connectors  installed  so  as  to  facilitate  access;  and  verification 
that  the  software  design  for  maintenance  and  diagnostic  routines 
and  procedures  are  adequate  and  that  the  software  can  be  modified 
to  accommodate  functional  changes. 

o  Continuity  of  Operations.  IOT&E  should  provide  for  an  impact 
assessment  of  the  failure  of  any  subsystem  element  of  a  C&C  system 
on  overall  mission  effectiveness. 

o  Imitative  Deception.  IOT&E  should  provide  for  tests  to  assess 
the  susceptibility  of  the  data  links  of  a  C&C  system  to  imitative 
deception. 
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o  Demonstration  of  Procedures.  Test  plans  should  include  a 
procedural  demonstration  whereby  the  tested  C&C  system  works  in 
conjunction  with  other  systems. 

o  Government  Furnished  Equipment  and  Facilities.  T&E  should  be 
concerned  about  the  availability  of  GFE  equipment  as  specified  in 
the  proposed  contract. 

o  User  Participation  in  T&E.  The  varying  needs  of  the  user  for 
a  C&C  system  make  it  mandatory  that  he  participate  in  all  phases 
of  T&E. 


25.7.3.4  Full  Rate  Production/Deployment  Phase 

o  First  Article  Testing.  Conduct  first  article  testing.  The 
preproduction,  first  article,  testing  and  evaluation  should  be  designed 
and  conducted  to:  (1)  confirm  the  adequacy  of  the  equipment  to  meet 
specified  performance  requirements;  (2)  confirm  the  adequacy  of  the 
software  not  only  to  meet  current  user  needs  but  also  to  accommodate 
changing  needs;  and  (3)  determine  failure  modes  and  rates  of  the 
total  integrated  system.  This  activity  should  be  followed  by  FOT&E. 

o  Test  Planners  and  Evaluators.  Use  the  IOT&E  personnel  in  the 
Follow-on  OT&E  program.  The  planners  and  evaluators  for  the  OT&E 
of  the  production  system  can  do  a  better  job  if  they  are  initially 
involved  in  planning  and  conducting  the  IOT&E. 

25.7.4  Ship  Systems 


25.7.4.1  Concept  Exploration/Definition  Phase 

o  Test  and  Evaluation  Master  Plan.  Prior  to  Milestone  I,  sufficient 
materiel  should  be  generated  to  allow  for  an  evaluation  of  the  overall 
T&E  program. 

o  Test  Objectives  and  Critical  Issues.  In  evaluating  the  initial 
test  concept,  it  is  important  that  the  test  objectives  during  the 
time  from  Milestone  I  to  Milestone  II  address  the  major  critical 
issues,  especially  technological  issues. 

o  OT&E  Phasing.  In  evaluating  test  plans,  look  favorably  on  phasing 
where  the  OT&E  is  run  in  parallel  with  continued  DT&E. 

o  Test  Facilities  and  Instrumentation  Required.  Before  Milestone 
I,  the  test  facilities  and  instrumentation  requirements  to  conduct 
developmental  and  operational  tests  should  be  identified,  along  with 
a  tentative  schedule  of  test  activities. 
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o  Multiple  Approach  To  Weapon  System  Development.  Whenever 
possible,  the  weapon  system  concept  should  not  be  predicated  on  the 
successful  development  of  a  single  hardware  or  software  approach 
in  the  various  critical  subsystems  (unless  it  has  been  previously 
demonstrated  adequately). 

o  Comparison  of  New  Versus  Old  System.  The  procedure  for  examining 
the  relative  performance  of  new  or  modified  systems  versus  old  should 
be  indicated  in  the  T&E  plan. 

o  Test  Support  Facilities.  The  phasing  of  test  support  facilities 
must  be  carefully  planned,  with  some  schedule  flexibility  to  cover 
late  delivery  and  other  unforeseen  problems. 

o  Fleet  Operating  Force  Requirements.  The  requirement  for  fleet 
operating  forces  for  DT&E  or  OT&E  should  be  assessed  early  in  the 
program,  and  a  specific  commitment  made  as  to  the  types  of  units 
to  be  employed. 

o  Mission  Related  Measures  of  Effectiveness.  During  the  Concept 
Exploration/Definition  Phase  of  the  acquisition  of  a  new  class  of 
ship,  a  study  effort  should  be  commenced  jointly  by  the  CNO  and 
COMOPTEVFOR  to  establish  mission-related  measures  of  effectiveness 
which  may  be  expressed  in  numerical  fashion  and  which  may  later  be 
made  the  subject  of  OT&E  to  determine  how  closely  the  new  ship  system 
meets  the  operational  need  for  which  it  was  conceived. 

o  Ship  T&E  Management.  The  management  of  ship  T&E  should  ensure 
that  test  requirements  are  necessary  and  consistent  relative  to 
systems/subsystem  aspects  and  that  the  necessary  testing  is  coordinated 
so  that  test  redundancy  does  not  become  a  problem. 

o  T&E  of  Large,  Integrally-Constructed  Systems.  Major  subsystems 
should  be  proven  feasible  prior  to  firm  commitment  to  a  detailed 
hull  design. 

25.7.4.2  Concept  Demonstration/Validation  Phase 

o  Authentication  of  Human  Factors  Concepts.  T&E  should  authenticate 
the  human  factors  concepts  embodied  in  the  proposed  systems  design, 
examining  questions  of  safety,  comfort,  appropriateness  of  man-machine 
interfaces,  as  well  as  the  numbers  and  skill  levels  of  the  personnel 
requi red. 

o  Acquisition  Strategy.  The  acquisition  strategy  for  a  ship  and 
its  subsystems  should  allow  for  a  sufficient  time  between  the  planned 
end  of  demonstration  testing  and  major  procurement  decisions  of 
government  furnished  equipment  so  that  there  is  a  flexibility  for 
modification  of  plans  which  may  be  required  during  the  test  phases 
of  the  program. 
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o  Evaluation  of  Results  of  Exploratory  Testing.  Results  of  tests 
conducted  during  exploratory  development  and  which  most  likely  have 
been  conducted  on  brassboards,  breadboards,  or  modified  existing 
hardware  should  be  carefully  evaluated. 

o  Software  Testing.  In  view  of  increased  dependence  upon  computers 
in  ship  management  and  tactical  operation,  software  testing  must 
be  exceptionally  thorough,  and  integrated  software  testing  must  begin 
as  early  as  possible. 

o  New  Hull  Forms.  When  a  new  type  of  ship  involves  a  radical 
departure  from  the  conventional  hull  form,  extensive  prototype  testing 
prior  to  further  commitment  to  the  new  hull  form  should  be  required. 

o  Effects  of  Hull  and  Propulsion  on  Mission  Capability.  The 
predicted  effects  of  the  proven  hull  and  propulsion  system  design 
on  the  performance  of  the  ship's  critical  system  should  be  determined. 

o  Advances  in  Propulsion.  Demonstration  of  the  use  of  new 

propulsion  systems  should  be  conducted  prior  to  making  the  decision 
to  commit  the  propulsion  systems  to  the  ship  in  question. 

o  Propulsion  Systems  in  Other  Classes.  When  an  engine  to  be  used 
in  the  propulsion  system  of  a  new  ship  is  already  performing 
satisfactorily  in  another  ship,  this  is  not  to  be  taken  as  an 
indication  that  shortcuts  can  be  taken  in  propulsion  system  DT&E, 
or  that  no  problems  will  be  encountered. 

o  IOT&E  of  Shipboard  Gun  Systems.  Operational  tests  of  shipboard 
gun  systems  should  simulate  the  stress,  exposure  time  and  other 
conditions  of  battle  so  that  the  suitability  of  the  weapon  can  be 
evaluated  in  total. 

o  Targets  for  Anti-Aircraft  Warfare  (AAW)  IOT&E.  Operational 

test  of  shipboard  AAW  weapons  demands  the  use  of  targets  which 
realistically  simulate  the  present  day  threat. 

o  Waivers  to  T&E  of  Ship  Systems.  Waivers  to  T&E  of  pre-production 
models  of  a  system  in  order  to  speed  up  production  and  delivery  should 
be  made  only  after  consideration  of  all  costs  and  benefits  of  the 
waiver,  including  those  not  associated  with  the  contract. 

o  Environment  Effects  on  Sonar  Domes.  Environmental  effects  on 

sonar  domes  and  their  self-noise  should  be  tested  and  evaluated  before 
the  domes  are  accepted  as  part  of  the  sonar  system. 

o  Hull/Machinery  Testing  By  Computer  Simulation.  In  DT&E  ships, 

there  will  be  cases  where  the  best  means  to  conduct  evaluations  of 


25-24 


particular  hull  and  machinery  capabilities  is  through  dynamic  analysis 
using  computer  simulation,  with  later  validation  of  the  simulation 
by  actual  test. 

o  Operational  Reliability.  IOT&E  should  provide  valuable  data 
on  the  operational  reliability  of  ship  weapon  systems  which  cannot 
be  obtained  through  DT&E. 

25.7.4.3  Full  Scale  Development  Phase 

o  Initial  or  Pilot  Phase  of  IOT&E.  Before  any  operational  tests 
for  demonstration  of  operational  suitability  and  effectiveness  are 
conducted,  an  initial  or  pilot  test  should  be  conducted. 

o  Identify  Critical  Subsystems.  In  the  planning  for  the  IOT&E 
of  a  ship  system,  the  critical  subsystems  with  respect  to  mission 
performance  should  be  identified. 

o  Reliability  of  Critical  Systems.  T&E  should  determine  the 
expected  reliability  at  sea  of  systems  critical  to  the  ship's  mobility 
and  primary  and  major  secondary  tasks. 

o  Consistency  in  Test  Objectives.  There  are  various  phases  of 
testing  of  a  ship  system.  One  should  ensure  that  the  objectives 
of  one  phase  are  not  inconsistent  with  the  objectives  of  the  other 
phases. 

o  Single  Screw  Ships.  T&E  of  the  propulsion  systems  of  ships 
with  a  single  screw  should  be  especially  rigorous  to  determine  failure 
rates,  maintenance  and  repair  alternatives. 

o  Problems  Associated  With  New  Hulls.  Whenever  a  new  hull  is 
incorporated  into  the  ship  design,  a  test  and  evaluation  of  this 
hull  should  be  conducted  prior  to  the  full-rate  production  and 
incorporation  of  the  major  weapons  subsystems. 

25.7.4.4  Full-Rate  Production  Phase 

o  Design  of  Ship  FOT&E.  In  the  testing  program  of  a  ship  system, 
it  should  be  recognized  that  although  it  may  be  designated  as  a  special 
purpose  ship,  it  will  in  most  cases  be  used  in  a  general  purpose 
role  as  well. 

o  Operational  Testing  During  Shakedown  Periods.  The  time  period 
for  OT&E  of  a  ship  can  be  used  more  efficiently  if  full  advantage 
is  taken  of  the  periods  immediately  after  the  ship  is  delivered  to 
the  Navy. 
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o  Fleet  Operations  in  FOT&E.  A  great  deal  of  information  on  the 
operational  effectiveness  of  a  ship  can  be  obtained  from  standard 
fleet  operations  through  well  designed  information  collection, 
processing,  and  analysis  procedures. 

o  Ship  ASW  OT&E  PLanning.  In  planning  OT&E  of  shipboard  systems, 
it  is  important  to  recognize  the  difficulty  of  achieving  realism, 
perhaps  more  so  than  in  other  areas  of  naval  warfare. 

o  Variable  Depth  Sonar  OT&E.  The  behavior  of  towed  bodies  of 
variable  depth  sonar  systems  and  towed  arrays  should  be  tested  and 
evaluated  under  all  ship  maneuvers  and  speeds  likely  to  be  encountered 
in  combat. 

o  Ship  Self-Noise  Tests.  The  magnetic  and  acoustic  signatures 
of  a  ship  can  be  tested  accurately  only  after  it  is  completed. 

o  Effect  of  Major  ECM  on  Ship  Capability.  The  FOT&E  of  a  ship 
should  include  tests  of  the  effectiveness  of  the  ship  when  subjected 
to  major  ECM. 

o  Ship  System  Survivability.  Operational  Test  and  Evaluation 
of  modern  ships  should  provide  for  the  assessment  of  their  ability 
to  survive  and  continue  to  fight  when  subjected  to  battle  damage. 

o  Interlocks.  Shipboard  electronic  systems  are  designed  with 
interlock  switches  that  open  electriral  circuits  for  safety  reasons 
when  the  equipment  cabinets  are  opened.  OT&E  should  be  able  to  detect 
over-design  as  well  as  minimum  design  adequacy  of  the  interlock 
systems. 

o  Intra-Ship  Communication.  In  the  conduct  of  lead  ship  trials 
and  evaluations,  particular  attention  should  be  given  to  the 
operational  impact  resulting  from  absence,  by  design,  of  intra-ship 
communications  circuits  and  stations  from  important  operating 
locations. 

25.7.5  Surface  Vehicle  Systems 

25.7.5.1  Concept  Exploration/Definition  Phase 

o  Preparation  of  Test  Plans.  It  is  necessary  that  a  detailed 
evaluation  criteria  be  established  which  includes  all  items  that 
are  to  be  tested. 

o  Validation  Test  Plans.  Prior  to  Milestone  I,  a  plan  should 
be  prepared  for  an  evaluation  of  the  overall  T&E  program.  As  part 
of  this,  a  detailed  test  and  evaluation  plan  for  those  tests  to  be 
conducted  prior  to  Milestone  II  to  validate  the  concept  and  hardware 
approach  to  the  vehicle  system  should  be  developed.  The  objective 
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of  the  validation  test  plan  is  to  fully  evaluate  the  performance 
characteristics  of  the  new  concept  vehicle.  This  test  plan  cannot 
be  developed,  of  course,  until  the  performance  characteristics  are 
defined. 

o  Performance  Characteristics  Range.  Stated  performance 
characteristics  derived  from  studies  should  be  measured  early  in 
the  program.  Unrealistic  performance  requirements  can  lead  to  false 
starts  and  costly  delays. 

o  Operating  Degradation.  System  performance  degrades  under  field 
conditions.  Anticipated  degradation  must  be  considered  during  test 
and  evaluation.  When  a  system  must  operate  at  peak  performance  during 
DT/OT  to  meet  the  specified  requirements  it  then  will  likely  perform 
at  a  lesser  level  when  operated  in  the  field. 

o  Test  Personnel.  The  Test  Director  and/or  key  members  of  the 
test  planning  group  within  the  project  office  should  have  significant 
T&E  experience. 

o  Design  Reviews.  T&E  factors  and  experience  must  influence  the 
system  design.  The  application  of  knowledge  derived  from  past 
experience  can  be  a  major  asset  in  arriving  at  a  sound  system  design. 

o  Prototype  Vehicles.  When  high  technical  risk  is  present, 
development  should  be  structured  around  the  use  of  one  or  more 
prototype  vehicles  designed  to  prove  the  system  concept  under  realistic 
operational  conditions  before  proceeding  to  engineering  development. 

o  Test  Facilities  and  Scheduling.  Before  Milestone  I,  test  range 
and  resource  requirements  to  conduct  validation  tests  should  be 
identified  along  with  a  tentative  schedule  of  test  activities. 


25.7.5.2  Concept  Demonstration/Validation  Phase 

o  Vulnerability.  The  vulnerability  of  vehicles  should  be  estimated 
on  the  basis  of  testing. 

o  Gun  and  Ammunition  Performance.  Gun  and  ammunition  development 
should  be  considered  a  part  of  overall  tank  system  development. 
When  a  new  gun  tube,  or  one  which  has  not  previously  been  mounted 
on  a  tank  chassis,  is  being  evaluated,  all  ammunition  types  (including 
missiles)  planned  for  use  in  that  system  should  be  test  fired  under 
simulated  operational  conditions. 

o  Increased  Complexity.  The  addition  of  new  capabilities  to  an 
existing  system  or  system  type  will  generally  increase  complexity 
of  the  system,  and  therefore  increase  the  types  and  amount  of  testing 
required  and  the  time  to  perform  these  tests. 


25-27 


o  Component  Interfaces.  Prior  to  assembly  in  a  prototype  system, 
component  subsystems  should  be  assembled  in  a  mockup  and  verified 
for  physical  fit,  human  factors  considerations,  interface 
compatibility,  and  electrical  and  mechanical  compatibility. 

o  Determination  of  Test  Conditions.  Test  conditions  during 
validation  should  be  determined  by  the  primary  objectives  of  that 
test,  rather  than  by  more  general  considerations  of  realism. 

o  Test  Plan  Development.  The  test  plan  developed  by  this  point 
should  be  in  nearly  final  form,  and  include  as  a  minimum: 

A  description  of  requirements. 

The  facilities  needed  to  make  evaluations. 

The  schedule  of  evaluations  and  facilities. 

The  reporting  procedure,  the  objective  of  which  is  to 
communicate  test  results  in  an  understandable  format  to 
all  program  echelons. 

The  Test  and  Evaluation  Guidelines,  and 

A  further  refinement  of  the  cost  estimates  which  were 
initiated  during  the  conceptual  phase. 

o  Demonstration  Tests.  Demonstration  tests  should  show  satisfactory 
meeting  of  success  criteria  which  are  meaningful  in  terms  of 
operational  usage.  In  designing  contractually  required  demonstration 
tests,  upon  whose  outcome  may  depend  large  incentive  payments,  or 
even  program  continuation,  it  is  essential  to  specify  broader  success 
criteria  than  simply  hit  or  miss  in  a  single  given  scenario. 

o  Reliability  Testing.  Reliability  testing  should  be  performed 
on  component  and  subsystem  assemblies  prior  to  testing  of  the  complete 
vehicle  system.  Prior  to  full  system  testing  viable  component  and 
subsystem  tests  should  be  conducted. 

o  Human  Factors.  In  evaluating  ground  vehicles,  human  factors 
should  be  considered  at  all  stages  starting  with  the  design  of  the 
prototype. 

o  Test  Plan  Scheduling.  Test  plan  scheduling  should  be  tied  to 
event  milestones  rather  than  to  the  calendar.  In  evaluating  the 
adequacy  of  the  scheduling  as  given  by  test  plans,  it  is  important 
that  milestones  be  tied  to  the  major  events  of  the  weapon  system 
(meeting  stated  requirements)  and  not  the  calendar. 
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o  Test  Failures.  The  T&E  schedule  should  be  sufficiently  flexible 
to  accommodate  failures  and  correction  of  problems  which  have  been 
identified. 

25.7.5.3  Full-Scale  Development  Phase 

o  Planning  the  Operational  Test.  Operational  testing  should  be 
cost  effective  and  provide  meaningful  results. 

o  Pilot  and  Dry  Run  Tests.  A  scheduled  series  of  tests  should 
be  preceded  by  a  dry  run  which  verifies  that  the  desired  data  will 
be  obtained. 

o  Comparison  Testing.  The  test  program  should  include  a  detailed 
comparison  of  the  characteristics  of  a  new  vehicle  system  with  those 
of  existing  systems,  alternate  vehicle  system  concepts  (if  applicable), 
and  those  of  any  system(s)  being  replaced. 

o  Simulations.  Simulation  techniques  and  equipment  should  be 
utilized  to  enhance  data  collection.  Creation  of  histograms  for 
each  test  course  provides  a  record  of  conditions  experienced  by  the 
vehicle  during  testing.  Use  of  a  chassis  dynamometer  can  produce 
additional  driveline  endurance  testing  with  more  complete 
instrumentation  coverage. 

o  Environmental  Testing.  Ground  vehicles  should  be  tested  in 
environmental  conditions  and  situations  comparable  to  those  in  which 
they  will  be  expected  to  perform. 

o  System  Vulnerability.  For  combat  vehicles,  some  estimate  of 
vulnerability  to  battle  damage  should  be  made. 

o  Design  Criteria  Verification.  Subsystem  design  criteria  should 
be  compared  with  actual  characteristics. 

o  Electromagnetic  Testing.  Vehicle  testing  should  include 
electromagnetic  testing. 

o  System  Strength  Testing.  In  evaluating  ground  vehicles,  early 
testing  should  verify  intrinsic  strength.  This  implies  operation 
with  maximum  anticipated  loading,  including  trailed  loads  at  maximum 
speeds  and  over  worst  case  grades,  secondary  roads,  and  cross-country 
conditions  for  which  the  vehicle  was  developed  or  procured.  This 
test  is  intended  to  identify  deficient  areas  of  design,  not  to  break 
the  machinery. 

o  Component  Compatibility.  Component  compatibility  should  be 
checked  through  the  duration  of  the  test  sequence. 
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o  Human  Interface.  Critiques  of  good  and  bad  features  of  the 
vehicle  should  be  made  early  in  the  prototype  stage,  while  adequate 
time  remains  to  make  any  indicated  changes. 

o  Serviceability  Testing.  Ground  vehicles  should  be  tested  and 
evaluated  to  determine  the  relative  ease  of  serviceability, 
particularly  with  high  frequency  operations. 

o  Experienced  User  Critique.  Ground  vehicle  user  opinions  should 
be  obtained  early  in  the  development  program. 

o  Troubleshooting  During  Tests.  Provisions  should  be  made  to 
identify  subsystem  failure  causes.  Subsystems  may  exhibit  failures 
during  testing.  Adequate  provisions  should  be  made  to  permit 
troubleshooting  and  identification  of  defective  components  and 
inadequate  design. 

25.7.5.4  Full-Rate  Production/Deployment  Phase 

o  Performance  and  Reliability  Testing.  The  production  first-article 
testing  should  verify  the  performance  of  the  vehicle  system  and 
determine  the  degradation,  failure  modes,  and  failure  rates. 

o  Lead-the-Fleet  Testing.  At  least  one  production  prototype  or 
initial  production  model  vehicle  should  be  allocated  in  intensive 
testing  so  as  to  accumulate  very  high  operating  time  in  a  short  period. 

o  User  Evaluation.  User-reported  shortcomings  should  be  followed 
up  to  determine  problem  areas  requiring  correction. 
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ABBREVIATIONS  AND  ACRONYMS 

Army  Acquisition  Executive 

Army  Development  and  Acquisition  Threat  Simulators 

Acquisition  Decision  Memorandum 

Air  Force  Operational  Test  &  Evaluation  Center 

Air  Force  Systems  Command 

Army  Materiel  Command 

Army  Material  Acquisition  Review  Committee 
Army  Materiel  Systems  Analysis  Activity 
Automatic  Test  Equipment 
Built-In  Test 
Command  and  Control 

Command,  Control,  Communications,  Intelligence 

Critical  Design  Review 

Contract  Data  Requirements  List 

Congressional  Data  Sheets 

Concept  Demonstration  /  Validation  Phase 

Concept  Exploration  /  Definition  Phase 

Contract  Line  Item  Number 

Candidate  Nomination  Proposal 

Cost  and  Operational  Effectiveness  Analysis 

Commander,  Operational  Test  and  Evaluation  Force 

Computer  Software  Component 

Developing  Agency  (Navy) 

Computer  Software  Configuration  Item 
Defense  Acquisition  Board 
Defense  Acquisition  Executive 
Decision  Coordinating  Paper 

Deputy  Director,  Defense  Research  &  Engineering  (Test 

&  Evaluation)  » 

Design  Limit  Test 

Data  Link  Vulnerability  Analysis 

DOD  Product  Engineering  Services  Office 

Director,  Operational  Test  and  Evaluation 
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DOE 

Department  of  Energy 

DRB 

Defense  Resources  Board 

DJ 

Development  Test 

DT&E 

Development  Test  &  Evaluation 

EA 

Evolutionary  Acquisition 

ECM 

Electronic  Countermeasures 

ECR 

Engineering  Change  Review 

EDT 

Engineering  Design  Test 

ERAM 

Extended  Range  Anti-armor  Munition 

ESM 

Electronic  Support  Measures 

EW 

Electronic  Warfare 

FCA 

Functional  Configuration  Audit 

FDT&E 

Force  Development  Tests  and  Experimentation 

FOT&E 

Follow-on  Test  and  Evaluation 

FORSCOM 

Forces  Command 

FQR 

Formal  Qualification  Review 

FSD 

Full-Scale  Development  Phase 

FWE 

Foreign  Weapons  Evaluation 

FYTP 

Five  Year  Test  Program 

GPMO 

Government  Program  Management  Office 

IEP 

Independent  Evaluation  Plan 

IFPP 

Information  for  Proposal  Preparation 

ILS 

Integrated  Logistics  Support 

ILSMT 

Integrated  Logistic  Support  Management  Team 

ILSP 

Integrated  Logistics  Support  Plan 

IOC 

Initial  Operating  Capability 

IOT&E 

Initial  Operational  Test  &  Evaluation 

IRA 

Industrial  Resource  Analysis 

ITP 

Integrated  Test  Plan 

IV&V 

Independent  Verification  and  Validation 

•JLF 

Joint  Live  Fire 

JRD 

Joint  Requirements  Document 

JT&E 

Joint  Test  and  Evaluation 

LFT 

Live  Fire  Test 

LRIP 

Low  Rate  Initial  Production 

MAA 

Mission  Area  Analysis 

MCCR 

Mission  Critical  Computer  Resources 

MCOTEA 

Marine  Corps  Operational  Test  &  Evaluation  Activity 

MMOU 

Multinational  Memorandum  of  Understanding 

MNS 

Mission  Needs  Statement 

MOE 

Measure  of  Effectiveness 

MOU 

Memorandum  of  Understanding 

MS 

Milestone 

MRTFB 

Major  Range  and  Test  Facility  Base 

NAPMA 

North  Atlantic  Program  Management  Agency 

NAVA I R 

Naval  Air  Systems  Command 

NAVSEA 

Naval  Sea  Systems  Command 

NBC 

Nuclear,  Biological,  and  Chemical 

NDCP 

Navy  Decision  Coordinating  Paper 

NDI 

Nondevelopment  Item 

OPNAV 

Operational  Navy 

OPEVAL 

Operational  Evaluation 

OPSEC 

Operations  Security 

OPTLi/FOR 

Operational  Test  &  Evaluation  Force 

OR 

Operational  Requirement 

ORMAS/TE 

Operational  Resource  Mgmt  Assessment  System  for  T&E 

OT 

Operational  Test 

OTA 

Operational  Test  Agency 

OTD 

Operational  Test  Director 

OT&E 

Operational  Test  &  Evaluation 

OTEA 

Operational  Test  &  Evaluation  Agency 

OTO 

Operational  Test  Organization 

OTP 

Outline  Test  Plan 

OSD 

Office  of  the  Secretary  of  Defense 

P3I 

Preplanned  Product  Improvements 

PAT&E 

Production  Acceptance  Test  and  Evaluation 

A- 3 


PCA 

Physical  Configuration  Audit 

PCO 

Primary  Contracting  Officer 

PDR 

Preliminary  Design  Review 

PDSS 

Post-Deployment  Software  Support 

P  MO 

Program  Management  Office 

POM 

Program  Objectives  Memorandum 

PPBS 

Planning,  Programming,  and  Budgeting  System 

PPQT 

Preproduction  Qualification  Tests 

PRESINSURV 

President  of  the  Boards  of  Inspection  and  Survey 

PRR 

Production  Readiness  Review 

QOT&E 

Qualification  Operational  Test  and  Evaluation 

RAM 

Reliability,  Availability,  and  Maintainability 

RDT 

Reliability  Development  Testing 

RDT&E 

Research,  Development,  Test  &  Evaluation 

RFP 

Request  for  Proposal 

ROC 

Required  Operational  Capability 

SAR 

Selected  Acquisition  Report 

SQA 

Software  Quality  Assurance 

SECARMY 

Secretary  of  the  Army 

SECDEF 

Secretary  of  Defense 

SECNAV 

Secretary  of  the  Navy 

SSR 

Software  Specification  Review 

SEF 

Stability  Enhancement  Function 

SIS 

Stall  Inhibit  System 

SON 

Statement  of  Operational  Need 

SOW 

Statement  of  Work 

SPO 

System  Program  Office 

SCP 

System  Concept  Paper 

SDR 

System  Design  Review 

SEMP 

System  Engineering  Management  Plan 

SRR 

Systems  requirements  Review 

STP 

Software  Test  Plan 

T4E 

Test  4  Evaluation 

TAAF 

Test,  Analyze  and  Fix 

PCA 

Physical  Configuration  Audit 

PCO 

Primary  Contracting  Officer 

PDR 

Preliminary  Design  Review 

PDSS 

Post-Deployment  Software  Support 

P  MO 

Program  Management  Office 

POM 

Program  Objectives  Memorandum 

PPBS 

Planning,  Programming,  and  Budgeting  System 

PPQT 

Preproduction  Qualification  Tests 

PRESINSURV 

President  of  the  Boards  of  Inspection  and  Survey 

PRR 

Production  Readiness  Review 

QOT&E 

Qualification  Operational  Test  and  Evaluation 

RAM 

Reliability,  Availability,  and  Maintainability 

ROT 

Reliability  Development  Testing 

RDT&E 

Research,  Development,  Test  &  Evaluation 

RFP 

Request  for  Proposal 

ROC 

Required  Operational  Capability 

SAR 

Selected  Acquisition  Report 

SQA 

Software  Quality  Assurance 

SECARMY 

Secretary  of  the  Army 

SECDEF 

Secretary  of  Defense 

SECNAV 

Secretary  of  the  Navy 

SSR 

Software  Specification  Review 

SEF 

Stability  Enhancement  Function 

SIS 

Stall  Inhibit  System 

SON 

Statement  of  Operational  Need 

SOW 

Statement  of  Work 

SPO 

System  Program  Office 

SCP 

System  Concept  Paper 

SDR 

System  Design  Review 

SEMP 

System  Engineering  Management  Plan 

SRR 

Systems  requirements  Review 

STP 

Software  Test  Plan 

T&E 

Test  &  Evaluation 

TAAF 

Test,  Analyze  and  Fix 

TEAM 

Test,  Evaluation,  Analysis,  and  Modeling 

TEC 

Test  and  Evaluation  Committee 

TECG 

T&E  Coordinating  Group 

TECOM 

Test  and  Evaluation  Command 

TEMP 

Test  and  Evaluation  Master  Plan 

TIWG 

Test  Integrated  Working  Group 

TMC 

Test  Management  Council 

TPO 

Test  Program  Outline 

TPWG 

Test  Planning  Working  Group 

TRMS 

TRADOC  Resource  Management  System 

TRADOC 

Training  and  Doctrine  Command 

TRR 

Test  Readiness  Review 

TSARC 

Test  Schedule  and  Review  Committee 

US  D ( A ) 

Under  Secretary  of  Defense  (Acquisition) 
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APPENDIX  B 
DOD  GLOSSARY  OF 
TEST  TERMINOLOGY 


ACCEPTANCE  TRIALS  -  Trials  and  material  inspection  conducted  underway 
by  the  trail  board  for  ships  constructed  in  a  private  shipyard  to  determine 
suitability  for  acceptance  of  a  ship. 

ACCRUED  EXPENDITURES  -  Costs  incurred  during  a  given  period  representing 
liabilities  incurred  for  goods  and  services  received,  other  assets  acquired, 
and  performance  accepted,  whether  or  not  payment  has  been  made. 

ACQUISITION  -  The  process  consisting  of  planning,  designing,  producing, 
and  distributing  a  weapon  system/equipment.  Acquisition  in  this  sense 
includes  the  conceptual,  validation,  full-scale  development,  production, 
and  deployment/operational  phases  of  the  weapon  systems/equipment  project. 
For  those  weapon  systems/equipments  not  being  procured  by  a  project  manager, 
it  encompasses  the  entire  process  from  inception  of  the  requirement  through 
the  operational  phase. 

ACQUISITION  CATEGORY  (ACAT)  -  One  of  four  acquisition  categories  established 
by  CNO  which  govern  acquisition  procedures  and  responsibilities  and  assign 
respective  decision  authority  levels. 

ACQUISITION  RISK  -  The  change  that  some  elements  of  an  acquisition  program 
produces  an  unintended  result  with  adverse  effect  on  system  effectiveness, 
suitability,  cost,  or  availability  for  deployment. 

ADVANCED  DEVELOPMENT  (Budget  Category  6.3)  -  Includes  all  projects  which 
have  moved  into  the  development  of  hardware  for  test. 

AGENCY  COMPONENT  -  A  major  organizational  subdivision  of  an  agency.  For 
example:  the  Army,  Navy,  Air  Force,  and  Defense  Supply  Agency  are  agency 
components  of  the  Department  of  Defense.  The  Federal  Aviation,  Urban 
Mass  Transportation,  and  the  Federal  Highway  Administrations  are  agency 
components  of  the  Department  of  Transportation. 

ALLOCATION  -  An  authorization  by  a  designated  official  of  a  component 
of  the  Department  of  Defense  making  funds  available  within  a  prescribed 
amount  to  an  operating  agency  for  the  purpose  of  making  allotments;  i.e., 
the  first  subdivision  of  an  apportionment. 

ANALYSIS  -  The  qualitative  and/or  quantified  evaluation  of  information 
requiring  technical  knowledge  and  judgement. 

APPORTIONMENT  -  A  determination  by  the  Office  of  Management  and  Budget 
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as  to  the  amount  of  obligations  which  may  be  incurred  when  the  nature 
of  the  work  involved  prevents  the  preparation  of  definitive  requirements, 
specifications,  or  cost  data.  Sometimes  called  letter  of  intent. 

AUTHORIZATION  -  Basic  substantive  legislation  enacted  by  Congress  which 
sets  up  a  Federal  program  or  agency  either  indefinitely  or  for  a  given 
period  of  time.  Such  legislation  sometimes  sets  limits  cn  the  amount 
that  can  subsequently  be  appropriated,  but  does  not  usually  provide  budget 
authority. 

AUTOMATIC  TEST  EQUIPMENT  (ATE)  -  An  equipment  that  is  designed  to 
automatically  conduct  analysis  of  functional  or  static  parameters  and 
to  evaluate  the  degree  of  performance  degradation  and  perform  fault 
isolation  of  unit  malfunctions. 

BASELINE,  APPROVED  -  The  combination  of  approved  program  schedule, 
configuration,  performance  characteristics,  acquisition,  strategy,  and 
other  business  aspects  which  constitute  the  variables  reflected  in  either 
the  appropriate  acquisition  milestone  approval  for  that  acquisition  category 
or  as  reflected  in  the  latest  approved  program  management  proposal  action. 

BRASSBOARD  CONFIGURATION  -  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  and  operational 
data.  It  will  normally  be  a  model  sufficiently  hardened  for  use  outside 
of  laboratory  environments  to  demonstrate  the  technical  and  operational 
principles  of  immediate  interest.  It  may  resemble  the  end  item  but  is 
not  intended  for  use  as  the  end  item. 

BREADBOARD  CONFIGURATION  -  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  data.  It  will 
normally  be  configured  only  for  laboratory  use  to  demonstrate  the  technical 
principles  of  immediate  interest.  It  may  not  resemble  the  end  item  and 
is  not  intended  for  use  as  the  projected  end  item. 

BUDGET  -  A  planned  program  for  a  fiscal  period  in  terms  of  (a)  estimated 
costs,  obligations  and  expenditures,  (b)  source  of  funds  for  financing, 
including  reimbursements  anticipated  and  other  resources  to  be  applied, 
and  (c)  explanatory  and  workload  data  on  the  projected  programs  and 
activities. 

CONCEPT  EVALUATION  PROGRAM  -  A  specifically  funded  Army  innovative  testing 
program.  CEP's  provide  commanders  and  combat  developers  a  quick  reaction 
and  simplified  process  to  resolve  combat  development,  doctrinal,  and 
training  issues  in  addition  to  solidifying  combat  development  requirements 
and  supporting  early  milestone  decisions.  Also,  the  CEP  is  used  to  provide 
an  experimental  data  base  for  requirements  documents  and  to  expedite  the 
materiel  acquisition  process;  however,  CEP's  are  not  to  be  used  as  the 
primary  tests  to  support  decision  review  production  decisions.  CEP  may 
be  conducted  at  any  time  to  support  the  CE  process.  Issues  satisfied 
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during  the  conduct  of  a  CEP  need  not  be  examined  during  formal  OT  to 
minimize  testing.  Data  from  CEP's  may  be  used  as  another  source  for 
preparation  of  the  IER. 

CONTINUOUS  COMPREHENSIVE  EVALUATION  (C2E)  -  A  continuous  process  extending 
from  concept  definition  through  deployment  which  evaluates  the  operational 
effectiveness  and  suitability  of  a  system  by  analysis  of  all  available 
data. 

COMBAT  SYSTEM  -  The  equipment,  computer  programs,  people  and  documentation 
organic  to  the  accomplishment  of  the  mission  of  an  aircraft,  surface  ship, 
or  submarine;  excludes  the  structure,  material,  propulsion,  power  and 
auxiliary  equipment,  transmissions  and  propulsion,  fuels  and  control 
systems,  and  silencing  inherent  in  the  construction  and  operation  of 
aircraft,  surface  ships  and  submarines. 

CONFIGURATION  MANAGEMENT  -  A  discipline  applying  technical  and 
administrative  direction  and  surveillance  to  (1)  identify  and  document 
the  functional  and  physical  characteristics  of  a  configuration  item,  (2) 
control  changes  to  those  characteristics,  and  (3)  record  and  report  change 
processing  and  implementation  status. 

CONTRACT  -  An  agreement,  enforceable  by  law,  between  two  or  more  competent 
parties,  to  do  or  not  to  do  something  not  prohibited  by  law,  for  a  legal 
consideration. 

CONTRACTOR  SUPPORT  -  An  arrangement  during  initial  development  or  production 
of  end-items  whereby  a  contractor  furnished  required  material  and 
maintenance  of  an  end-item  or  system  pending  assumption  of  supply  by  the 
military  service. 

COST  OPERATIONAL  EFFECTIVENESS  ANALYSIS  -  A  method  of  examining  alternative 
means  of  accomplishing  a  desired  military  objective/mission  for  the  purpose 
of  selecting  weapons  and  forces  which  will  provide  the  greatest  military 
effectiveness  for  the  cost. 

CRITICAL  ISSUES  -  Those  aspects  of  a  system's  capability  either  operational, 
technical,  or  other,  that  must  be  questioned  before  a  system's  overall 
worth  can  be  estimated,  and  that  are  of  primary  importance  to  the  decision 
authority  in  reaching  a  decision  to  allow  the  system  to  advance  into  the 
next  acquisition  phase. 

DATA  SYSTEM  -  Combinations  of  personnel  efforts,  forms,  formats, 
instructions,  procedures,  data  elements  and  related  data  codes, 
communications  facilities,  and  automatic  data  processing  equipment,  which 
provide  an  organized  and  interconnected  means,  either  automated,  manual, 
or  a  mixture  of  these  for  recording,  collecting,  processing  and 
communicating  data. 


DEFENSE  ACQUISITION  EXECUTIVE  (DAE)  -  The  principal  advisor  to  the  Secretary 
of  Defense  on  all  matters  pertaining  to  the  Department  of  Defense 
Acquisition  System.  Tlie  Under  Secretary  of  Defense  for  Acquisition 
•(US  D(A))  is  the  DAE  and  the  Defense  Procurement  Executive. 

DEPARTMENT  OF  DEFENSE  ACQUISITION  SYSTEM  -  A  single  uniform  system  whereby 
all  equipment,  facilities,  and  services  are  planned,  designed,  developed, 
acquired,  maintained,  and  disposed  of  within  the  Department  of  Defense, 
the  system  entails  establishing  policies  and  practices  that  govern 
acquisitions,  determining  and  prioritizing  resource  requirements,  directing 
and  controlling  the  process,  contracting,  and  reporting  to  Congress. 

DESIGNATED  ACQUISITION  PROGRAM  -  Program  designated  by  the  Army  Acquisition 
Executive  for  DRB  milestone  review. 

DESIGNATED  OPERATIONAL  TESTER  -  The  Major  Command  or  Army  Special  Staff 
Agency  delegated  authority  to  conduct  specific  OT.  Designated  operational 
testers  conduct  the  operational  test  of  assigned  systems  and  will  normally 
prepare  the  evaluation  of  that  system.  The  actual  operational  test  and/or 
evaluation  is  usually  accomplished  by  a  subordinate  element  of  the 
designated  operational  tester.  The  designated  operational  tester  can 
be  TRADOC,  USAISC,  TSG,  INSCOM,  COE,  or  another  designated  command  or 
agency. 

DEMONSTRATION  AND  VALIDATION  DECISION  -  Milestone  I  decision  by  which 
the  SECDEF  reaffirms  the  mission  need  and  approves  one  or  more  selected 
alternatives  for  competitive  demonstration  and  validation. 

DEVELOPING  AGENCY  (DA)  -  The  Systems  Command  or  CNM-designated  project 
manager  assigned  responsibility  for  the  development,  test  and  evaluation 
of  a  weapon  system,  subsystem  or  item  of  equipment. 

DEVELOPER  EVALUATION  -  The  developer's  evaluation  addresses  all  aspects 
of  the  system  to  include  technical  performance,  operational  effectiveness, 
and  operational  suitability  cost  and  schedule. 

DEVELOPMENT  TEST  -  A  technical  test  conducted  post-MS  I,  pre-MS  II  to 
provide  data  on  safety,  the  achievabil ity  of  critical  system  technical 
characteristics,  refinement  and  ruggedization  of  hardware  configurations, 
and  determination  of  technical  risks.  This  testing  is  performed  on 
components,  subsystems,  materiel  improvement,  nondevelopment  items  ( ND I ) , 
hardware-software  integration  and  related  software.  DT  includes  the  testing 
of  compatibility  and  interoperability  with  existing  or  planned  equipment 
and  systems  and  the  system  effects  caused  by  natural  and  induced 
environmental  conditions  during  the  development  phases  of  the  materiel 
acquisition  process.  Program  funding  category  6.3. 

ENGINEERING  CHANGE  PROPOSAL  -  Proposal  to  change  design  or  engineering 
features  of  materiel  under  development  or  production.  Includes  proposed 
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engineering  change  and  documentation  by  which  the  change  is  described 
and  suggested. 

ENGINEERING  DEVELOPMENT  -  RDTE  funding  category  that  includes  development 
programs  being  engineered  for  Service  use,  but  not  yet  approved  for 
procurement  or  operation. 

EFFECTIVENESS  -  The  performance  or  output  received  from  an  approach  or 
a  program.  Ideally,  it  is  a  quantitative  measure  which  can  be  used  to 
evaluate  the  level  of  performance  in  relation  to  some  standard,  set  of 
criteria,  or  end  objective. 

ENGINEERING  CHANGE  -  An  alteration  in  the  physical  or  functional 
characteristics  of  a  system  or  item  delivered,  to  be  delivered,  or  under 
development,  after  establishment  of  such  characteristics. 

ENGINEERING  DEVELOPMENT  (Budget  Category  6.4)  -  Includes  those  projects 
in  full-scale  development  of  Service  use  but  which  have  not  yet  received 
approval  for  production  or  had  production  funds  included  in  the  DoD  budget 
submission  for  the  budget  or  subsequent  fiscal  year. 

EVALUATION  CRITERIA  -  Standards  by  which  achievement  of  required  technical 
and  operational  effectiveness/suitability  characteristics,  or  resolution 
of  technical  or  operational  issues,  may  be  evaluated.  At  Milestone  I 
and  II,  evaluation  criteria  should  include  quantitative  thresholds  for 
the  IOC  system.  At  Milestone  III  and  beyond  (or  IOC,  whichever  occurs 
first),  evaluation  criteria  should  include  quantitative  thresholds  for 
the  mature  system.  If  system  maturity  is  greater  than  2  years  beyond 
IOC,  intermediate  evaluation  criteria,  appropriately  time-lined,  must 
also  be  provided. 

FIVE-YEAR  DEFENSE  PROGRAM  -  The  official  document  which  summarizes  the 
SECDEF-approved  plans  and  programs  for  the  Department  of  Defense.  It 
is  published  at  least  once  annually. 

FOLLOW-ON  OPERATIONAL  TEST  AND  EVALUATION  -  Test  and  evaluation  conducted 
subsequent  to  a  full  production  decision  to  obtain  information  lacking 
from  previous  Operational  Test  and  Evaluation,  or  to  verify  correction 
of  materiel,  training,  or  concept  deficiencies. 

FOLLOW-ON  PRODUCTION  TEST  -  A  technical  test  conducted  subsequent  to  a 
full  production  decision  on  initial  production  and  mass  production  models 
to  determine  production  conformance  for  quality  assurance  purposes.  Program 
funding  category  -  Procurement. 

GOAL  -  Something  to  which  one  aspires  for  a  program,  or,  a  point  aimed 
at  for  achievement. 
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INTEGRATED  LOGISTIC  SUPPORT  (ILS)  -  A  disciplined,  unified,  and  Iterative 
approach  to  the  management  and  technical  activities  necessary  to  :  (a) 
integrate  support  considerations  into  system  and  equipment  design;  (b) 
develop  support  requirements  that  are  related  consistently  to  readiness 
objectives,  to  design,  and  to  each  other,  (c)  acquire  the  required  support; 
and  (d)  provide  the  required  support  during  the  operational  phase  at  minimum 
cost. 

INTEROPERABILITY  -  The  ability  of  systems,  units,  or  forces  to  provide 
services  to,  and  accept  services  from,  other  systems,  units  or  forces, 
and  to  use  the  services  so  exchanged  to  enable  them  to  operate  together 
effectively. 

INDEPENDENT  EVALUATION  REPORT  -  A  report  that  provides  an  assessment  of 
item  or  system  operational  effectiveness  and  operational  suitability  versus 
critical  issues  as  well  as  the  adequacy  of  testing  to  that  point  in  the 
development  of  item  or  system. 

INDEPENDENT  TEST  AGENCY  -  The  Army  Operational  Test  and  Evaluation  Agency, 
the  Navy  Operational  Test  and  Evaluation  Force,  the  Air  Force  Operational 
Test  and  Evaluation  Center,  and  the  Marine  Corps  Operational  Test  and 
Evaluation  Agency. 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  -  The  first  phase  of  operational 
test  and  evaluation  conducted  on  preproduction  items,  prototypes,  or  pilot 
production  items  and  normally  completed  prior  to  the  first  major  production 
decision.  It  is  conducted  to  provide  a  valid  estimate  of  expected  system 
operational  effectiveness  and  suitability. 

IN-PROCESS  REVIEW  -  Review  of  a  project  or  program  at  critical  poii.s 
to  evaluate  status  and  make  recommendations  to  the  decision  authority. 
Conducted  by  the  MATDEV. 

ISSUES  -  Any  aspect  of  the  system's  capability,  either  operational, 
technical,  or  other,  that  must  be  questioned  before  the  system's  overall 
military  utility  can  be  known.  Operational  issues  are  those  that  must 
be  evaluated  considering  the  soldier  and  the  machine  as  an  entity  to 
estimate  the  operational  effectiveness,  and  operational  suitability  of 
the  system  in  its  complete  user  environment. 

JOINT  DEVELOPMENT  TESTS  -  JDT  provides  information  on  intraservice  system 
or  equipment  requirements,  performance,  or  interoperability;  on  technical 
concepts,  requirements,  or  improvements;  and  on  the  improvement  or 
development  of  testing  methodologies  or  resources. 

JOINT  OPERATIONAL  TESTS  -  JOT  uses  actual  fielded  equipment,  simulators, 
or  surrogate  equipment  in  an  exercise  or  operational  environment  to  obtain 
data  pertinent  to  interservice  operational  doctrine,  tactics,  and 
procedures. 
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LETHALITY  -  The  probility  that  weapon  effects  will  destroy  the  target 
or  render  it  neutral. 

LIFE-CYCLE  COST  -  The  total  cost  to  the  Government  for  the  development, 
acquisition,  operation  and  logistic  support  of  a  system  or  set  of  forces 
over  a  defined  life  span. 

LOGISTICS  SUPPORTABILITY  -  The  degree  of  which  the  planned  logistics  support 
(including  test  equipment,  measurement  and  diagnostic  equipment,  spares 
and  repair  parts,  technical  data,  support  facilities,  transportation 
requirements,  training  and  manpower)  allow  meeting  system  availability 
and  wartime  usage  requirements. 

LONG  LEAD  ITEMS  -  Those  components  of  a  system  or  piece  of  equipment  that 
take  the  longest  time  to  procure,  and  therefore,  may  require  an  early 
commitment  of  funds  in  order  to  meet  acquisition  program  schedules. 

LOW  RATE  INITIAL  PRODUCTION  -  Any  manufacture  of  a  system  in  limited 
quantity  to  be  used  in  OT&E  for  verification  of  production  engineering 
and  design  maturity  and  to  establish  a  production  base. 

MAINTAINABILITY  -  A  characteristic  of  design  and  installation  which  is 
expressed  as  the  probability  that  an  item  will  be  retained  in  or  restored 
to  a  specified  condition  within  a  given  period  of  time,  when  the  maintenance 
is  performed  in  accordance  with  prescribed  procedures  and  resources. 

MAJOR  DEFENSE  ACQUISITION  PROGRAM  -  As  specified  in  10  United  Stated  Code, 
Sections  136a  and  139a  (reference  (1))  and  DoD  Directive  5000.1  (reference 
(m)). 

a.  A  DoD  acquisition  program  that  is  not  a  highly 
sensitive/classified  program  (as  determined  by  the  Secretary  of  Defense) 
and: 

(1)  That  is  designated  by  the  Secretary  of  Defense  as  a  major 
defense  acquisition  program;  or 

(2)  That  is  estimated  by  the  Secretary  of  Defense  to  require 
an  eventual  total  expenditure  for  research,  development,  test  and  evaluation 
of  more  than  200  million  dollars  (based  on  fiscal  year  1980  constant 
dollars)  or  an  eventual  total  expenditure  for  procurement  of  more  than 
1  billion  dollars  (based  on  fiscal  year  1980  constant  dollars). 

b.  A  DoD  acquisition  program  that  is  designated  jointly  by  the 
DOT&E  and  DDDRE(T&E),  as  a  major  defense  acquisition  program  for  the  purpose 
of  carrying  out  the  responsibilities,  functions,  and  authorities  of  this 
Manual.  Such  designation  for  the  purpose  of  Test  and  Evaluation  oversight 
does  not  imply  any  other  related  review  requirements. 


MAJOR  RANGE  AND  TEST  FACILITY  BASE  (MRTFB)  -  The  complex  of  major  DoD 
ranges  and  test  facilities. 

MILESTONE  -  A  major  management  decision  point  in  the  overall  acquisition 
process  of  a  major  DoD  system  requiring  OSD  and/or  DoD  Component  program 
review.  Milestones  include  both  Joint  Resource  Management  Board  (DRB) 
and  DoD  Component  equivalent  Program  Reviews. 

MILITARY  REQUIREMENT  -  An  established  need  justifying  the  timely  allocation 
of  resources  to  achieve  a  capability  to  accomplish  approved  military 
objectives,  missions,  or  tasks.  Requirements  are  normally  documented 
in  a  MSR,  0&0  Plan,  LOA,  ROC,  LR,  TDLR,  TDR  or  SN-CIE. 

MISSION  AREA  ANALYSIS  -  Continuous  analysis  of  assigned  mission 
responsibilities  in  the  several  mission  areas  to  identify  deficiencies 
in  the  current  and  projected  capabilities  to  meet  essential  mission  needs, 
and  to  identify  opportunities  for  the  enhancement  of  capability  through 
more  effective  systems  and  less  costly  methods. 

MISSION  NEED  STATEMENT  -  Submitted  prior  to  POM  submission.  Approval 
by  SECDEF  is  Milestone  0.  Documents  major  mission  deficiencies  (or 
opportunities  for  improvement)  in  a  service's  ability  to  meet  mission 
requirements  when  such  deficiencies  can  be  corrected  by:  (1)  using  an 
existing  U.S.  system  or  allied  military  or  commercial  system,  (2)  a  major 
modification  to  an  existing  system,  or  (3)  a  new  major  acquisition.  A 
joint  MNS  is  prepared  to  document  major  deficiencies  in  two  or  more  DoD 
components.  OSD  or  OJCS  may  also  prepare  MNS. 

MISSION  RELIABILITY  -  The  probability  that  the  system  will  perform  mission 
essential  functions  for  a  period  of  time  under  the  conditions  stated  in 
the  mission  profile. 

MODEL  -  A  model  is  a  representation  of  an  actual  or  conceptual  system 
that  involves  mathematics,  logical  expressions,  or  computer  simulations 
that  can  be  used  to  predict  how  the  system  might  perform  or  survive  under 
various  conditions  or  in  a  range  of  hostile  environment. 

MULTISERVICE  OPERATIONAL  TEST  -  A  form  of  test  when  one  or  more  of  the 
services  provide  support  service  test  or  vice  versa,  or  tests  that  involve 
agreements  between  a  service  and  one  or  more  of  the  other  services  to 
evaluate  a  system  or  concept  that  requires  testing  in  a  multiservice 
environment. 

NUCLEAR  HARDNESS  -  A  quantitative  description  of  the  physical  attributes 
of  the  system  or  component  that  will  allow  nuclear  survivability  in  a 
given  weapon  environment.  Hardness  is  measured  by  physical  quantities 
such  as  overpressure,  peak  velocities,  energy  absorbed,  electrical  stress, 
etc.  Hardness  is  achieved  through  design  specifications  and  often  verified 
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by  one  or  more  test  and  analysis  techniques.  Hardness  is  only  one  of 
several  means  of  attaining  system-wide  nuclear  survivability. 

NONDEVELOPMENT  ITEM  (NDI)  -  Already  developed  and  available  hardware  and/or 
software  capable  of  fulfilling  Service  requirements,  thereby  minimizing 
or  eliminating  the  need  for  costly,  time-consuming  Government-sponsored 
R&D  programs.  NDI  is  usually  off-the-shelf  or  commercial -type  products, 
but  may  also  include  equipment  already  developed  by  or  for  the  military 
services  or  foreign  military  forces. 

OPERATIONAL  AVAILABILITY  (Ao)  -  An  index  of  a  weapon  system  materiel 
readiness,  including  system  software  where  applicable,  in  a  mission 
environment.  It  is  a  measure  of  the  probability  of  an  item's  being  in 
a  condition,  generally  referred  to  as  "up",  such  that  it  can  perform  its 
intended  function,  within  acceptable  limits  of  degradation,  when  called 
upon. 

OPERATIONAL  EFFECTIVENESS  -  The  overall  degree  of  mission  accomplishment 
of  a  system  used  by  representative  personnel  in  the  context  of  the 
organization,  doctrine,  tactics,  threat  (including  countermeasures  and 
nuclear  threats),  and  environment  in  the  planned  operational  employment 
of  the  system. 

OPERATIONAL  EVALUATION  -  Addresses  the  effectiveness  and  suitability  of 
the  weapons,  equipment,  or  munitions  for  use  in  combat  by  typical  military 
users  and  the  system  operational  issues  and  criteria;  provides  information 
to  estimate  organizational  structure,  personnel  requirements,  doctrine, 
training  and  tactics;  identifies  any  operational  deficiencies  and  the 
need  for  any  modifications;  and  assesses  MANPRINT  (safety,  health  hazards, 
human  factors,  manpower  and  personnel)  aspects  of  the  system,  in  a  realistic 
operational  environment. 

OPERATIONAL  REQUIREMENT  (OR)  -  The  basic  requirement  document  for  all 
Navy  acquisition  programs  requiring  research  and  development  effort. 

OPERATIONAL  SUITABILITY  -  The  degree  to  which  a  system  can  be  placed 
satisfactorily  in  field  use,  with  consideration  being  given  to  availability, 
compatibility,  transportability,  interoperability,  reliability,  wartime 
usage  rates,  maintainability,  safety,  human  factors,  manpower 
supportability,  logistic  supportabil ity,  and  training  requirements. 

OPERATIONAL  TEST  AND  EVALUATION  (OTSE)  -  The  field  test  under  realistic 
combat  conditions,  of  any  item  (or  key  component  of)  weapons,  equipment, 
or  munitions  for  the  purpose  of  determining  the  effectiveness  and 
suitability  of  the  weapons,  equipment,  or  munitions  for  use  in  combat 
by  typical  military  users;  and  the  evaluation  of  the  results  of  such  test. 


OPERATIONAL  TEST  CRITERIA  -  Expressions  of  the  operational  level  of 
performance  required  of  the  military  system  to  demonstrate  operational 
effectiveness  for  given  functions  during  each  operational  test.  The 
expression  consists  of  the  function  addressed,  the  basis  for  comparison, 
the  performance  required,  and  the  confidence  level. 

OPERATIONAL  TEST  READINESS  REVIEW  -  A  review  to  identify  problems  that 
may  impact  the  conduct  of  an  OT&E.  OTRR's  are  conducted  to  determine 
changes  required  in  planning,  resources,  or  testing  necessary  to  proceed 
with  the  OT-OTRR  participants  include  the  operational  tester  (chair), 
evaluator,  material  developer,  user  representative,  logisticians,  HQDA 
staff  elements  and  others  as  necessary. 

OPERATIONAL  TEST  -  Testing  of  materiel  systems  that  is  accomplished  with 
representative  user  operators,  crews,  support  personnel,  or  units  in  as 
realistic  an  operational  environment  as  possible  to  provide  the  evaluator 
data  to  estimate: 

a.  The  military  operational  effectiveness,  and  operational 
suitability  (including  compatibility,  interoperability,  reliability, 
availability,  and  maintainability,  supportabi 1 i ty ,  operational 
soldier/hardware/software  interface,  and  training  requirements)  of 
new  systems. 

b.  The  system's  desirability,  from  the  use  viewpoint,  considering 
systems  already  available  and  the  operational  benefits  and/or  burdens 
associated  with  the  new  system. 

c.  The  need  for  modification  to  the  system. 

d.  The  adequacy  of  doctrine,  organization,  operating  techniques, 
tactics,  and  training  for  employment  of  the  system;  the  adequacy 
of  maintenance  and  supply  support  for  the  system;  the  adequacy  of 
maintenance  and  supply  support  for  the  system;  and,  when  appropriate, 
its  performance  in  a  countermeasure  environment. 

PILOT  PRODUCTION  -  The  controlled  manufacture  of  limited  numbers  of  an 
item  for  service  test  and  evaluation  purposes  using  manufacturing  drawings 
and  specifications  which  have  been  developed  for  quantity  production  and 
with  tooling  that  is  representative  of  that  to  be  used  in  unlimited 
production. 

POST-PRODUCTION  TESTIN6  -  Testing  conducted  to  assure  that  materiel  which 
is  reworked,  repaired,  renovated,  rebuilt,  or  overhauled  after  initial 
issue  and  deployment  conforms  to  specified  quality,  reliability,  safety, 
and  operational  performance  standards.  Included  in  post-production  tests 
are  surveillance  tests,  stockpile  reliability,  and  reconditioning  tests. 
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PREPLANNED  PRODUCT  IMPROVEMENT  -  Planned  future  evolutionary  improvement 
of  developmental  systems  for  which  design  considerations  are  effected 
during  development  to  enhance  future  application  of  projected  technology. 
Includes  improvements  planned  for  ongoing  systems  that  go  beyond  the  current 
performance  envelope  to  achieve  a  needed  operational  capability. 

PREPRODUCTION  PROTOTYPE  -  An  article  in  final  form  employing  standard 
parts,  representative  of  articles  to  be  produced  on  a  production  line 
with  production  tooling. 

PREPRODUCTION  QUALIFICATION  TEST  -  The  formal  contractual  tests  that  ensure 
design  integrity  over  the  specified  operational  and  environmental  range. 
These  tests  usually  use  prototype  or  preproduction  hardware  fabricated 
to  the  proposed  production  design  specifications  and  drawings.  Such  tests 
include  contractural  reliability  and  maintainability  demonstrations  tests 
required  prior  to  production  release. 

PRODUCT  IMPROVEMENT  PLAN  (PIP)  -  Effort  to  incorporate  a  configuration 
change  involving  engineering  and  testing  effort  on  end  items  and  depot 
repairable  components,  or  changes  on  other  than  developmental  items  to 
increase  system  or  combat  effectiveness  or  extend  the  useful  military 
life. 

PRODUCTION  PROVE  OUT  TESTS  -  A  technical  test  conducted  post  MS  II  or 
post  MS  I  / 1 1  prior  to  production  testing  with  prototype  hardware.  This 
testing  provides  data  on  safety,  the  achievabil ity  of  critical  system 
technical  characteristics,  refinement  and  ruggedization  of  hardware 
configurations,  and  determination  of  technical  risks.  After  type 
classification,  production  prove  out  testing  may  also  be  conducted  to 
provide  data  which  could  not  be  obtained  prior  to  technical  compliance, 
such  as  survivability  or  environmental.  Program  funding  category  -  6.4. 

PRODUCTION  QUALIFICATION  TEST  -  A  technical  test  conducted  post-MS  III 
to  ensure  the  effectiveness  of  the  manufacturing  process,  equipment  and 
procedures.  This  testing  also  serves  the  purpose  of  providing  data  for 
the  independent  evaluation  required  for  materiel  release  so  that  the 
evaluator  can  address  the  adequacy  of  the  materiel  with  respect  to  the 
stated  requirements.  These  tests  are  conducted  on  a  number  of  samples 
taken  at  random  from  the  first  production  lot,  and  are  repeated  if  the 
process  or  design  is  changed  significantly,  and  when  a  second  or  alternative 
source  is  brought  on  line.  Program  funding  category  -  Procurement. 

PROGRAM  MANAGER  -  Individual  chartered  by  the  Service  Secretary  reporting 
to  the  material  developer  or  to  the  commander  of  a  subordinate  organization 
as  designated  by  the  material  developer.  Assigned  responsibility  and 
delegated  full-line  authority  of  the  material  developer  for  centralized 
management  of  a  specified  acquisition  or  materiel  readiness  program. 
May  be  superimposed  over  one  or  more  product  managers. 
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QUALITY  ASSURANCE  -  A  planned  and  systematic  pattern  of  all  actions 
necessary  to  provide  adequate  confidence  that  materiel  conforms  to 
established  technical  requirements  and  achieves  satisfactory  performance 
in  service. 

QUALIFICATIONS  TESTING  -  Which  verifies  the  design  and  manufacturing  process 
and  provides  a  baseline  for  subsequent  acceptance  tests.  The  completion 
of  Preproduction  Qualification  Test  and  Evaluation  before  MS  III  decisions 
Is  essential  and  will  be  a  critical  factor  in  assessing  the  system's 
readiness  for  production.  Production  Qualification  T&E  shall  be  conducted 
on  production  items. 

RELIABILITY  -  The  probability  that  an  item  will  perform  its  intended 
function  for  a  specified  Interval  under  stated  conditions. 

REALISTIC  TEST  ENVIRONMENT  -  The  conditions  under  which  a  system  is  expected 
to  be  operated  and  maintained,  including  the  natural  weather  and  climatic 
conditions,  terrain  effects,  battlefield  disturbances,  and  enemy  +hreat 
conditions. 

REQUIRED  OPERATIONAL  CAPABILITY  (ROC)  -  A  brief  statement  of  a  isoecific 
operational  capability  which  is  required  in  the  mid-range  period. 

RESEARCH  (Budget  Category  6.1)  -  Includes  all  effort  of  scientific  study 
and  experimentation  directed  toward  (1)  increasing  knowledge  and 
understanding  in  those  fields  of  the  physical,  engineering,  environmental 
and  life  sciences  related  to 'long-term  national  security  needs.  It  provides 
fundamental  knowledge  required  for  the  solution  of  military  problems. 
It  forms  a  part  of  the  base  for  (a)  subsequent  exploratory  and  advanced 
developments  in  Defense-related  technologies,  and  (b)  new  and  improved 
military  functional  capabilities  in  areas  such  as  communications,  detection, 
tracking,  surveillance,  propulsion,  mobility,  guidance  and  control, 
navigation,  energy  conversion,  materials  and  structures,  and  personnel 
support. 

RISK  -  An  expression  of  possible  loss  in  terms  of  hazard  severity  and 
hazard  probability. 

RISK  ASSESSMENT  -  An  evaluation  of  a  risk  in  terms  of  mission  loss  should 
a  hazard  result  in  an  accident. 

REQUIRED  OPERATIONAL  CHARACTERISTICS  -  Qualitative  and  quantitative  system 
parameters  approved  by  the  user  that  are  primary  indicators  of  a  system's 
capability  to  accomplish  its  mission  (operational  effectiveness)  and  to 
be  supported  (operational  suitability). 

REQUIRED  TECHNICAL  CHARACTERISTICS  -  Quantitative  system  parameters  approved 
by  the  DoD  Component  that  are  selected  as  primary  indicators  of  technical 
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achievement  of  engineering  thresholds.  These  might  not  be  direct  measures 
of,  but  should  always  relate  to,  a  system's  capability  to  perform  its 
required  mission  function  and  to  be  supported. 

SAFETY  -  Freedom  from  those  conditions  that  can  cause  death,  injury, 

occupational  illness,  or  damage  to,  or  loss  of,  equipment  or  property. 

SAFETY/HEALTH  VERIFICATION  -  The  development  of  data  used  to  evaluate 

the  safety  and  health  features  of  a  system  to  determine  its  acceptability. 
This  is  done  primarily  during  developmental  test  (DT)  and  user  or 

operational  test  (OT)  and  evaluation  and  supplemented  by  analysis  and 
independent  evaluations. 

SAFETY  RELEASE  -  A  formal  document  issued  to  a  user  test  organization 
before  any  hands-on  use  or  maintenance  by  personnel.  The  Safety  Release 
indicates  the  system  is  safe  for  use  and  maintenance  by  typical  user 

personnel  and  describes  the  system  safety  analyses.  Operational  limits 

and  precautions  are  included.  The  test  agency  uses  the  data  to  integrate 
safety  into  test  controls  and  procedures  and  to  determine  if  the  test 

objectives  can  be  met  within  these  limits. 

SELECTED  ACQUISITION  REPORT  -  Standard,  comprehensive,  summary  status 
report  on  DoD  acquisition  programs  for  management  within  DoD  provided 
to  Congress. 

SIMULATION  -  A  simulation  is  a  method  for  implementing  a  model,  it  is 

the  process  of  conducting  experiments  with  a  model  for  the  purpose  of 

understanding  the  behavior  of  the  system  modeled  under  selected  conditions 
or  limits  imposed  by  developmental  or  operational  criteria.  Simulation 
may  include  the  use  of  analog  or  digital  devices,  laboratory  models,  or 
"testbed"  sites.  Simulations  are  usually  programmed  for  solution  on  a 
computer;  however,  in  the  broadest  sense  military  exercises  and  wargames 
are  also  simulations. 

SPECIFICATION  -  A  specific  quantitative,  contractually  binding,  required 
operational  or  technical  characteristic. 

SUBTEST  -  An  element  of  a  test  program.  A  subset  is  a  test  conducted 
for  a  specific  purpose,  (e.g.,  rain,  dust,  transportabil ity,  missile  firing, 
fording) . 

SUITABILITY  -  A  subjective  determination  by  a  decision  authority  that 
a  materiel  system  does  or  does  not  meet  minimum  standards  prerequisite 
to  satisfy  field  service  use.  The  judgement  may  be  based  on  the  presence 
or  absence  of  uncorrectable  materiel  deficiencies,  and/or  the  number  and 
assessed  importance  of  correctable  and  uncorrectable  shortcomings.  It 
also  includes  judgements  on  nonmateriel  issues. 
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SURVIVABILITY  -  The  capability  of  a  system  to  avoid  or  withstand  man-made 
hostile  environments  without  suffering  an  abortive  impairment  of  its  ability 
to  accomplish  its  designated  mission. 

SUSCEPTIBILITY  -  The  degree  to  which  a  device,  equipment,  or  weapon  system 
is  open  to  effective  attack  due  to  one  or  more  inherent  weaknesses. 
(Susceptibility  is  a  function  of  operational  tactics,  countermeasures, 
probability  of  enemy  fielding  a  threat,  etc.). 

SYSTEM  -  A  composite,  at  any  level  of  complexity,  or  personnel,  procedures, 
materials,  tools,  equipment,  facilities,  and  software.  The  elements  of 
this  composite  entity  are  used  together  in  the  intended  operational  or 
support  environment  to  perform  a  given  task  or  achieve  a  specific 
production,  support,  or  mission  requirement. 

SYSTEM  ENGINEERING,  DEFENSE  -  That  portion  of  the  acquisition  process 
dealing  with  the  transformation  of  an  operational  need  into  an  optimal 
set  of  system  performance  parameters  and  a  preferred  system  configuration. 
It  includes  engineering/technical  management,  definition  of  system  and 
program,  design  engineering,  support  engineering,  the  integration  of  the 
engineering  specialties,  and  other  such  factors  that  affect  the  development, 
production,  deployment,  operation,  and  disposal  of  the  system. 

SYSTEM  ENGINEERING  PROCESS  -  A  logical  sequence  of  activities  and  decisions 
transforming  an  operational  need  into  a  description  of  system  performance 
parameters  and  a  preferred  system  configuration. 

TECHNICAL  EVALUATION  -  Addresses  the  system's  technical  issues  and  criteria 
and  the  acquisition  and  fielding  of  an  effective,  supportable,  and  safe 
system  by  assisting  in  the  engineering  design  and  development  and  verifying 
attainment  of  technical  performance  specifications,  objectives, 
producibil ity,  adequacy  of  the  Technical  Data  Package,  and  supportabil ity; 
determining  safety,  health  hazards,  human  factors,  and  MANPRINT  aspects. 
Technical  evaluation  encompasses  the  use  of  models,  simulations,  and 
testbeds,  as  well  a  prototypes  or  full-scale  development  models  of  the 
system. 

TECHNICAL  FEASIBILITY  TEST  -  A  technical  test  conducted  post-MS  0,  pre-MS 
I  or  MS  I  / 1 1  (under  the  Army  Streamlined  Acquisition  Process)  to  assist 
in  determining  safety  and  establishing  syste.  arformance  specifications 
and  feasibility. 

TECHNICAL  TESTER  -  The  command  or  agency  that  plans,  conducts,  and  reports 
the  results  of  the  Army  technical  testing.  Associated  contractors  may 
perform  development  testing  on  behalf  of  the  command  or  agency. 

TECHNICAL  TESTS  -  A  generic  term  for  testing  which  gathers  technical  data 
during  the  conduct  of  development  testing,  technical  feasibility  testing. 
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qualification  testing,  joint  development  testing,  and  contractor/foreign 
testing.  Soldier  operator-maintainer  test  and  evaluation  personnel  are 
used  during  the  conduct  of  technical  testing,  when  appropriate. 

TEST  CRITERIA  -  Standards  by  which  test  results  and  outcome  are  judged. 

TEST  AND  EVALUATION  MASTER  PLAN  -  An  overall  test  and  evaluation  plan, 
prepared  as  early  as  possible  in  the  acquisition  process,  and  is  designed 
to  identify  and  integrate  objectives,  responsibilities,  resources,  and 
schedule  for  all  test  and  evaluation  to  be  accomplished  prior  to  key 
decision  milestones. 

TESTBEDS  -  A  system  representation  consisting  partially  of  actual  hardware 
and  partially  of  computer  models  or  prototype  hardware. 

TEST  INSTRUMENTATION  -  Test  instrumentation  is  scientific,  ADPE,  or 
technical  equipment  used  to  measure,  sense,  record,  transmit,  process, 
or  display  data  during  tests,  evaluations,  or  examination  of  materiel, 
training  concepts,  or  tactical  doctrine.  Audio-visual  is  included  as 
instrumentation  when  used  to  support  Army  testing. 

TEST  RESOURCES  -  A  collective  term  that  encompasses  all  elements  necessary 
to  plan,  conduct  and  collect/analyze  data  from  a  test  event  or  program. 
Elements  include  test  funding  and  support  manpower  (including  TDY  costs), 
test  assets  (or,  units  under  test),  test  asset  support  equipment,  technical 
data,  simulation  models,  testbeds,  threat  simulators,  surrogates  and 
replicas,  special  instrumentation  peculiar  to  a  given  test  asset  or  test 
event,  targets,  tracking  and  data  acquisition,  instrumentation,  and 
equipment  for  data  reduction,  communications,  meteorology,  utilities, 
photography,  calibration,  security,  recovery,  maintenance  and  repair, 
frequency  management  and  control,  and  base/facility  support  services. 

TEST  DESIGN  PLAN  -  A  statement  of  the  conditions  under  which  the  test 
is  to  be  conducted,  the  data  required  from  the  test,  and  the  data  handling 
required  to  relate  the  data  results  to  the  test  conditions. 

THREAT  -  The  sum  of  the  potential  strength,  capabilities,  and  intentions 
of  an  enemy  which  can  limit  or  negate  mission  accomplishment  or  reduce 
force,  system,  or  equipment  effectiveness. 

THRESHOLDS  -  The  minimum  level  a  system  must  meet  (e.g.  performance 
threshold  of  30K  ft.  for  a  missile). 

TRANSPORTABILITY  -  The  inherent  capability  of  materiel  to  be  moved  by 
towing,  by  self-propulsion,  or  by  carrier  via  railways,  highways,  waterways, 
pipelines,  ocean,  and  airways. 
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USER  REPRESENTATIVE  -  The  combat  developer  designated  to  represent  the 
user  during  the  materiel  acquisition  process.  The  command  or  agency 
fulfilling  this  role  represents  the  "  mission-oriented"  user  and  the 
"logistics-oriented"  user  by  concerning  itself  with  both  the  operational 
and  logistic  support  aspects  of  materiel  system. 

VULNERABILITY  -  The  characteristics  of  a  system  that  cause  it  to  suffer 
a  definite  degradation  (loss  or  reduction  of  capability  to  perform  the 
designated  mission)  as  a  result  of  having  been  subjected  to  a  certain 
(defined)  level  of  effects  in  an  unnatural  (man-made)  hostile  environment. 

WORK  BREAKDOWN  STRUCTURE  -  A  product-oriented  family  tree  division  of 
hardware,  software,  services  and  other  work  tasks  which  organizer,  defines 
and  graphically  displays  the  product  to  be  produced  as  well  as  the  work 
to  be  accomplished  to  achieve  the  specified  product. 
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APPENDIX  D 
POINTS  OF  CONTACT 
SERVICE  TEST  &  EVALUATION  COURSES 


ARMY 

Commander  U.S.  Army  Test  and  Evaluation  Command 
ATTN :  AMSTE-TE 

Aberdeen  Proving  Ground,  Maryland  21005-5055 

Commander  U.S.  Army  Logistics  Management  College 

ATTN:  AMXMC-ACM-MA 

Ft.  Lee,  Virginia  23801-6048 

Commander  U.S.  Army  Operational  Test  &  Evaluation 
Agency 

ATTN:  CSTE-PO 

5600  Columbia  Pike 

Falls  Church,  Virginia  22041-5115 

NAVY 


Commander  Space  &  Naval  Warfare  Systems  Command 
ATTN:  Management  Operations 
National  Center,  Bldg.  1 
Washington,  D.C.  20361 

Commander  Operational  Test  &  Eval  ation  Force 

ATTN :  Dest  02B 

Norfolk,  Virginia  23511-6388 


AIR  FORCE 


Commander  Air  Force  Operational  Test  and 
Evaluation  Center 
ATTN:  Asst.  Director,  Training 
Building  20130 

Kirtland  Air  Force  Base,  New  Mexico  87117-7001 

Commander  Air  Force  Institute  of  Technology 
ATTN:  Student  Operations 
Wright-Patterson  Air  Force  Base,  Ohio  45433 
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